I want to send data from master database to replica database using Debezium and kafka. There are 300+ tables in the actual database and I want to replicate 15 of these tables.
I am using the{
“name”: “producer”,
“config”: {
“connector.class”: “io.debezium.connector.sqlserver.SqlServerConnector”,
“tasks.max”:“1”,
“database.hostname”: “hostname”,
“database.port”: “db_port”,
“database.user”: “username”,
“database.password”: “password”,
“database.server.name”: “server_name”,
“database.names”:“db_name”,
“table.include.list”: “tables i want to replicate”,
“schema.history.internal.kafka.bootstrap.servers”: “kafka:9092”,
“schema.history.internal.kafka.topic”: “schemahistory.fullfillment”,
“time.precision.mode”: “connect”,
“topic.prefix”:“topic_prefix”,
“transforms”: “route”,
“transforms.route.type”: “org.apache.kafka.connect.transforms.RegexRouter”,
“transforms.route.regex”: “topic_prefix\.db_name\.dbo\.(.*)”,
“transforms.route.replacement”: “$1”,
“database.encrypt”:“false”
}
} configuration file on the producer side.
On the consumer side, I use the {
“name”: “consumer”,
“config”: {
“connector.class”: “io.confluent.connect.jdbc.JdbcSinkConnector”,
“tasks.max”: “1”,
“connection.url”: “jdbc:sqlserver://hostname:port;databaseName=db_name;trustServerCertificate=true”,
“connection.user”: “user”,
“connection.password”: “password”,
“insert.mode”: “upsert”,
“delete.enabled”: “true”,
“auto.create”: “true”,
“table.name.format”:“${topic}”,
“topics”: “topics i want to replicate”,
“pk.mode”: “record_key”,
“pk.fields”: “id”,
“transforms”: “unwrap”,
“transforms.unwrap.type”: “io.debezium.transforms.ExtractNewRecordState”,
“transforms.unwrap.drop.tombstones”: “false”
}
} configuration file.
I am able to view the data in UI for Apache Kafka. But when transferring the data to the replica database, I get the error Invalid object name ‘The name of the table that is in the master database but not in the replica database, which I do not want to replicate’.
What is the solution for this error?
Also, when transferring image type data in MsSQL database, I encounter ‘nvarchar is incompatible with image’ error.
Why am I getting this error when I don’t include this table in the producer side?
Can anyone please help me?