The JDBC Sink connector streams data from Kafka to a relational database and relational databases have schemas The JDBC Sink connector therefore requires a schema to be present for the data.
The kind of errors that you can see as a side-effect of not providing a schema with your data include:
org.apache.kafka.connect.errors.ConnectException: Sink connector 'my-db-sink' is configured with 'delete.enabled=false' and 'pk.mode=none' and therefore requires records with a non-null Struct value and non-null Struct schema, but found record at (topic='nevergonnagiveyouup',partition=0,offset=0,timestamp=1626416123456) with a HashMap value and null value schema.
Depending on where your data is coming from you have different options.
- If it’s ingested from Kafka Connect, use a converter that supports schemas (Avro, Protobuf, JSON Schema)
- If it’s produced by an application that you have control over, get that application to serialise that data with a schema (Avro, Protobuf, JSON Schema)
- If it’s coming from somewhere you don’t have control over then you’ll need to pre-process the topic to add an explicit schema and write it to a new topic that is then consumed by the JDBC Sink connector.
References & resources: