Kafka self managed debezium parameters issue

Kafka Connect Self-Managed Connectors
1

Hi There,

Here is my error log(partial) as follows
curl -s “http://localhost:8083/connectors/source-debezium-orders-01/status” | jq ‘.’

{
“name”: “source-debezium-orders-01”,
“connector”: {
“state”: “RUNNING”,
“worker_id”: “connect-2:8084”
},
“tasks”: [
{
“id”: 0,
“state”: “FAILED”,
“worker_id”: “connect-2:8084”,
“trace”: "org.apache.kafka.connect.errors.ConnectException: Creation of database history topic failed, please create the topic manually\n\tat io.debezium.relational.history.KafkaDatabaseHistory.initializeStorage(KafkaDatabaseHistory.java:509)\n\tat io.debezium.relational.HistorizedRelationalDatabaseSchema.initializeStorage(HistorizedRelationalDatabaseSchema.java:69)\n\

My question is
How do I tag a topic to parameter “database.history.kafka.topic” while creating topic manually?

Regards
Abdul

1 Like
2

Hi, did you manage to solve this issue, I’m facing exactly the same error

3

A quick fix might be to create the topic manually (the schema.history.internal.kafka.topic for Debezium 2+ or database.history.kafka.topic for 1.9 and earlier).

Or, you could investigate why the admin client that attempts to create the topic (here for 2.6) is failing. Are there more details about the exception in the connect log?

What does your connector config look like? Specifically the properties prefixed by schema.history.internal. or database.history. as those are what get used to create the admin client. What kind of authentication is required for Kafka and do the schema.history.internal.producer.* or database.history.producer.* params look correct for authenticating?

1 Like
4

@dtroiano thanks for the response. Just find out what was the cause of my error.

I’m using MSK and EKS to deploy the connector, I was pre-creating manually the topics through Terraform but it turns out the IAM role I was using was lacking some permissions over the new topics due to I change the prefix.

What I did to fix was to add permissions:

"kafka-cluster:WriteData",
"kafka-cluster:ReadData",
"kafka-cluster:*Topic*"

And set the new topic prefix to the resources sections and that solved my issue.

1 Like