Hello all, we are having an issue deserializing an object value from a Kafka record in Kafka Connect. The record is not logging any errors that we can find via various debugs or ending up in the DLQ topic. It gets processed and sent to the sink connector where it is written to the sink, but this one value is an empty object “{}”. I thought the problem might be the sink’s ingestion process, so we wrote a custom SMT that logged the messages. In the SMT log, you can see that the key in question is empty at the point that the SMT transform is applied. I have turned on various debugs in Kafka Connect trying to get some idea as to why this item’s value is being dropped.
We’re using:
Confluent Cloud Kafka w/ schema registry
Self-Hosted CP Kafka Connect 7.7.1
io.confluent.connect.json.JsonSchemaConverter as the value converter
Here is a copy of the SMT log:
[2024-10-04 22:56:44,977] INFO Received record: key=00-eba9ff5004b3491185aee69496734de4-fae29596ef6e44d9-01, value=Struct{severity=6,component=vmanage_orchestrator,audience=9,org_primary_domain=xyz.com,tenant_slug=xyzdot4,data=Struct{},id=00-eba9ff5004b3491185aee69496734de4-fae29596ef6e44d9-01,message=Selected template sdwan_single_lte_site_dhcp for device FCWXXXXXX,timestamp=2024-10-04T22:53:07.794486Z}, topic=common-device-job-logs, partition=0, offset={} (com.xyz.kafka.connect.transforms.LoggingSMT)
The missing data should be in the value.data item. It is a JSON object that is fairly complex, but the details of that object are untyped in our schema registry.
“data”: {
“title”: “Data”,
“type”: “object”
}
Any ideas on how to further get to the root of this problem?