MirrorMaker2 High Availability

Kafka Connect
1

How are high availability and fault tolerance organized in MM2 ? Does it possible to run multiple instances of MM2 in Kafka connect to replicate same topic that would be created in different host ? My expectations if one worker goes down MM2 will continue replicating data using other workers.

I tried to create 3 instances of MM2 Connector to replicate one topic and specified same consumer group in parameter “consumer.group.id” but i’ve got duplicates in destination topic equals to number of MM2 Connector instances.

2

Do you have any tips related to the topic ?

3

I assume you’ve created 3 standalone instances?

could you share how you’ve started the stack?

best,
michael

4

I use this configuration to define MM2 connector workers on target side but with different “name” like my-mm2-conn-1, my-mm2-conn-2, my-mm2-conn-3

{
“name”: “my-mm2-conn-1”,
“config”: {
“consumer.group.id”: “mm2-group”,
“consumer.group.instance.id”: “mm2-grp-inst”,
“consumer.client.id”: “single-to-cluster-consumer”,
“connector.class”: “org.apache.kafka.connect.mirror.MirrorSourceConnector”,
“tasks.max”: “3”,
“source.cluster.alias”: “from_single”,
“target.cluster.alias”: “to_cluster”,
“target.cluster.bootstrap.servers”: “cluster-1:9092,cluster-2:9092,cluster-3:9092”,
“source.cluster.bootstrap.servers”: “single:9092”,
“topics”: “topic-a”,
“sync.topic.configs.enabled”: “false”,
“emit.heartbeats.enabled”: “true”,
“emit.checkpoints.enabled”: “true”,
“sync.group.offsets.enabled”: “true”,
“replication.factor”: “3”,
“heartbeats.topic.replication.factor”: “1”,
“checkpoints.topic.replication.factor”: “1”,
“offset-syncs.topic.replication.factor”: “1”
}
}

Then I create the connector using REST API

curl -X POST -H ‘Content-Type: application/json’ cluster-1:8083/connectors --data-binary @/data/home/user/my-mm2-conn.json

5

you don’t need to start 3 connectors for the same topic
(this is were you’re duplicates come from)
one connector should be sufficient and connect framework should take care of task failover etc

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.