Kafka Connector Debezium for SqlServer

We are trying to use the Sql Server connector to keep upstream Sql Server account data synced to downstream systems and the debezium connector sounded like just the tool for this…along with MQ connectors for some downstream systems… I’m fairly new to Kafka and I’ve been unable to find information about this error so it feels like an environment issue or Sql Server issue but I have been having a heck of a time with this error:

org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.\n\tat io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42)\n\tat io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:121)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: io.debezium.DebeziumException: io.debezium.DebeziumException: io.debezium.relational.history.DatabaseHistoryException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Topic dbhistory.exceptons not present in metadata after 10000 ms.\n\tat io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:82)\n\tat io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:105)\n\t… 5 more\nCaused by: io.debezium.DebeziumException: io.debezium.relational.history.DatabaseHistoryException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Topic dbhistory.exceptons not present in metadata after 10000 ms.\n\tat io.debezium.relational.RelationalSnapshotChangeEventSource.lambda$createSchemaChangeEventsForTables$2(RelationalSnapshotChangeEventSource.java:276)\n\tat io.debezium.pipeline.EventDispatcher.dispatchSchemaChangeEvent(EventDispatcher.java:276)\n\tat io.debezium.relational.RelationalSnapshotChangeEventSource.createSchemaChangeEventsForTables(RelationalSnapshotChangeEventSource.java:271)\n\tat io.debezium.relational.RelationalSnapshotChangeEventSource.doExecute(RelationalSnapshotChangeEventSource.java:127)\n\tat io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:71)\n\t… 6 more\nCaused by: io.debezium.relational.history.DatabaseHistoryException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Topic dbhistory.exceptons not present in metadata after 10000 ms.\n\tat io.debezium.relational.history.KafkaDatabaseHistory.storeRecord(KafkaDatabaseHistory.java:262)\n\tat io.debezium.relational.history.AbstractDatabaseHistory.record(AbstractDatabaseHistory.java:92)\n\tat io.debezium.relational.HistorizedRelationalDatabaseSchema.record(HistorizedRelationalDatabaseSchema.java:88)\n\tat io.debezium.connector.sqlserver.SqlServerDatabaseSchema.applySchemaChange(SqlServerDatabaseSchema.java:63)\n\tat io.debezium.pipeline.EventDispatcher$SchemaChangeEventReceiver.schemaChangeEvent(EventDispatcher.java:460)\n\tat io.debezium.relational.RelationalSnapshotChangeEventSource.lambda$createSchemaChangeEventsForTables$2(RelationalSnapshotChangeEventSource.java:273)\n\t… 10 more\nCaused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Topic dbhistory.exceptons not present in metadata after 10000 ms.\n\tat org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.(KafkaProducer.java:1307)\n\tat org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:962)\n\tat org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:862)\n\tat org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:750)\n\tat io.debezium.relational.history.KafkaDatabaseHistory.storeRecord(KafkaDatabaseHistory.java:247)\n\t… 15 more\nCaused by: org.apache.kafka.common.errors.TimeoutException: Topic dbhistory.exceptons not present in metadata after 10000 ms.\n

If anyone could nudge me in the right direction I would appreciate it very very much.

Also I’ll apologize in advance if I break any forum etiquette protocols.

Thank you.

Hi,
Pre-create the history topic with 1 partition before deploying your connector. According to logs your topic name should be dbhistory.exceptons.

In addition, i recommend you to read this 2 articles by @rmoff that would be useful for your use case
Streaming data from SQL Server to Kafka to Snowflake ❄️ with Kafka Connect

Using Kafka Connect and Debezium with Confluent Cloud