Cannot get JDBC SQL Server sink to work

Kafka Connect
1

Please help. Total noob but dont undertstand where im going wrong.

I am having issue Steaming from SQL Server ro SQL Server.

The Debezium source is working perfectly but when trying to write to SQL Server with JDBC whatever I try It fail. At one point it did create the id in a new table in a new DB then died again.

However I set the config for key and value conversion I get errors

#source config#

{
“name”: “sourceconn1”,
“config”: {
“connector.class” : “io.debezium.connector.sqlserver.SqlServerConnector”,
“tasks.max” : “1”,
“database.hostname” : “df-dev-mssql”,
“database.port” : “1433”,
“database.user” : “sa”,
“database.password” : “P@ssw0rd1999”,
“database.dbname” : “T014_rel-colotesting”,
“table.include.list”: “cas.Cases”,
“database.server.name” : “test7”,
“database.history.kafka.bootstrap.servers” : “localhost:9092”,
“database.history.kafka.topic”: “schema-changes7”
}
}

#sink config#

{
“name”: “sinkconn1”,
“config”:{
“connector.class”: “io.confluent.connect.jdbc.JdbcSinkConnector”,
“tasks.max”: “1”,
“connection.url”: “jdbc:sqlserver://df-dev-mssql:1433”,
“database”:“Sink-DB”,
“connection.user”: “sa”,
“connection.password”: “P@ssw0rd1999”,
“topics”: “test7”,
“delete.enabled”:“true”,
“pk.mode”:“record_key”,
“pk.field”:“id”,
“auto.create”: “true”

#ditributited props#

key.converter.schemas.enable=false
value.converter.schemas.enable=false

plugin.path=/home/kafka/kafka/plugins

Source is pure SQL CDC

2

Hi Gareth,

which errors do you get?

Best,
Michael

1 Like
3

Hi

Thanks for the reply, a bit more research and I found I needed to flatten JSON so for referernce for anyone new to Kafka I added these to source;

 "transforms":"unwrap",
 "transforms.unwrap.type":"io.debezium.transforms.ExtractNewRecordState",
 "transforms.unwrap.delete.handling.mode":"none",

and with a few changes to sink config:

"delete.enabled":"true",
"insert.mode":"upsert",

Have insert, update and delete working perfectly.

Now to play with transforms and kSQL.

Thanks again for the support.

Gareth

1 Like
4

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.