Replicator creates topic on source cluster

Kafka Connect Managed Connectors
1

Hello everybody,

I am currently trying to synchronize one topic from a kafka source cluster to a kafka target cluster. I am using the Confluent Replicator for this task but without the cloud service. Both clusters have a own schema registry.
This is my current configuration:

{
“connector.class”: “io.confluent.connect.replicator.ReplicatorSourceConnector”,
“tasks.max”: “1”,
“src.kafka.bootstrap.servers”: “docker-01:29093”,
“src.schema.registry.url”: “http://mq-schema-registry.docker-01:8081”,
“dest.kafka.bootstrap.servers”: “docker-02:29093”,
“dest.schema.registry.url”: “http://mq-schema-registry.docker-02:8081”,
“topic.whitelist”: “topic_xxx”,
“topic.rename.format”: “TEST_${topic}”,
“confluent.topic.replication.factor”: 1,
“key.converter”: “io.confluent.connect.avro.AvroConverter”,
“value.converter”: “io.confluent.connect.avro.AvroConverter”,
“key.converter.schema.registry.url”: “http://mq-schema-registry.docker-02:8081”,
“value.converter.schema.registry.url”: “http://mq-schema-registry.docker-02:8081”,
“schema.subject.translator.class”: “io.confluent.connect.replicator.schemas.DefaultSubjectTranslator”,
“src.key.converter”: “io.confluent.connect.avro.AvroConverter”,
“src.key.converter.schema.registry.url”: “http://mq-schema-registry.docker-01:8081”,
“src.value.converter”: “io.confluent.connect.avro.AvroConverter”,
“src.value.converter.schema.registry.url”: “http://mq-schema-registry.docker-01:8081”,
“offset.start”: “connect”,
“auto.offset.reset”: “earliest”
}

The sync of my schema from source to target works perfectly fine. But the topic itself makes some troubles. On the target cluster the topic gets created but there is no data synchronized. Additional to that the topic gets created on the source cluster too but with the data in it. Please help me to solve this issue.

Thanks and best regards!

2

hey @JulianG

both clusters are running on your local machine within docker correct?

anything in the logs?

3

Hey @mmuehlbeyer,

both clusters are running on separate servers.
The two servers can reach each other because they are in the same sub network.
Both clusters are running within docker.

In the logs I dont get any errors but a warning which says that no offsets could be found. DO you need the exact warning?

Thanks for your help!

4

This is ther warning I get:
WARN Could not find offset for group replicator, topic xxx, partition 0, timestamp 1707912236111, waiting for replication to catch up. Please check replication lag. (io.confluent.connect.replicator.offsets.ConsumerOffsetsTranslator)

5

hey @JulianG

sorry for the delay.
would you mind sharing the logs and your replicator config?

6

hey @mmuehlbeyer the replicator config is in my first post and the warning I get is in my third post.

Thanks for your help!

7

I see is there an existing consumer group on the topic you’d like to replicate?

8

No I just want to replicate the whole topic

9

Quick update:

Everything is working now exactly like I want it. The schema gets replicated, the topic gets created and the data is also replicated.
BUT there is one problem. All of this points happen on the source cluster. How is this possible?

10

I figured it out. My problem was the deployment…

I deployed the connector on the source side now it is on the destination is and it works.

Thanks for your help!

11

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.