The JDBC Sink Connector and Schemas

Kafka Connect
1

The JDBC Sink connector streams data from Kafka to a relational database and relational databases have schemas :smiley: The JDBC Sink connector therefore requires a schema to be present for the data.

The kind of errors that you can see as a side-effect of not providing a schema with your data include:

org.apache.kafka.connect.errors.ConnectException: Sink connector 'my-db-sink' is configured with 'delete.enabled=false' and 'pk.mode=none' and therefore requires records with a non-null Struct value and non-null Struct schema, but found record at (topic='nevergonnagiveyouup',partition=0,offset=0,timestamp=1626416123456) with a HashMap value and null value schema.

Depending on where your data is coming from you have different options.

  • If it’s ingested from Kafka Connect, use a converter that supports schemas (Avro, Protobuf, JSON Schema)
  • If it’s produced by an application that you have control over, get that application to serialise that data with a schema (Avro, Protobuf, JSON Schema)
  • If it’s coming from somewhere you don’t have control over then you’ll need to pre-process the topic to add an explicit schema and write it to a new topic that is then consumed by the JDBC Sink connector.

References & resources:

2

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.