How are high availability and fault tolerance organized in MM2 ? Does it possible to run multiple instances of MM2 in Kafka connect to replicate same topic that would be created in different host ? My expectations if one worker goes down MM2 will continue replicating data using other workers.
I tried to create 3 instances of MM2 Connector to replicate one topic and specified same consumer group in parameter “consumer.group.id” but i’ve got duplicates in destination topic equals to number of MM2 Connector instances.
Do you have any tips related to the topic ?
I assume you’ve created 3 standalone instances?
could you share how you’ve started the stack?
best,
michael
I use this configuration to define MM2 connector workers on target side but with different “name” like my-mm2-conn-1, my-mm2-conn-2, my-mm2-conn-3
{
“name”: “my-mm2-conn-1”,
“config”: {
“consumer.group.id”: “mm2-group”,
“consumer.group.instance.id”: “mm2-grp-inst”,
“consumer.client.id”: “single-to-cluster-consumer”,
“connector.class”: “org.apache.kafka.connect.mirror.MirrorSourceConnector”,
“tasks.max”: “3”,
“source.cluster.alias”: “from_single”,
“target.cluster.alias”: “to_cluster”,
“target.cluster.bootstrap.servers”: “cluster-1:9092,cluster-2:9092,cluster-3:9092”,
“source.cluster.bootstrap.servers”: “single:9092”,
“topics”: “topic-a”,
“sync.topic.configs.enabled”: “false”,
“emit.heartbeats.enabled”: “true”,
“emit.checkpoints.enabled”: “true”,
“sync.group.offsets.enabled”: “true”,
“replication.factor”: “3”,
“heartbeats.topic.replication.factor”: “1”,
“checkpoints.topic.replication.factor”: “1”,
“offset-syncs.topic.replication.factor”: “1”
}
}
Then I create the connector using REST API
curl -X POST -H ‘Content-Type: application/json’ cluster-1:8083/connectors --data-binary @/data/home/user/my-mm2-conn.json
you don’t need to start 3 connectors for the same topic
(this is were you’re duplicates come from)
one connector should be sufficient and connect framework should take care of task failover etc