Hello!
I’m wondering if it’s possible to run multiple instances of MM2 per data center? I am trying to run 2 instances of connect-mirror-maker.sh per data center for scalability purposes. However, I cannot get 2 instances to work at the same time: once I start up the second mirror maker, it takes over for the first one, and the first one completely stops replicating.
What am I missing here? Would greatly appreciate any guidance.
Thank you!
Hi @jk4102
welcome to the forum 
could you elaborate a bit more?
what would you like to achieve ? replicate the same topic with two independent mirror maker instances?
best,
michael
Hi @mmuehlbeyer, thanks for your response
I would like to replicate Kafka events between two data centers with two independent mirror maker instances, in a way that lets me load balance between the two mirror makers rather than have the two instances both do the same work. So, if I have my mirror makers replicating from A to B, I would like one instance to handle part of the produced events and the other instance to handle the rest. And if one mirror maker instance were to go down, the process would rebalance so that the remaining instance would pick up from where it left off and become responsible for all replication until the second instance becomes available again. That could mean that both mirror makers are replicating different parts of the same topic, but NOT that they are both replicating the whole topic
Thank you so much
hi @jk4102
ok understood.
basically it should be possible.
did you configure as the documentation states ?
https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP382:MirrorMaker2.0-RunningadedicatedMirrorMakercluster
is there some traffic going on the topics?
best,
michael
Yes, I’m using below configuration, and there is traffic in the topic.
mm2.properties config:
clusters = A, B
A.bootstrap.servers = localhost:9092
B.bootstrap.servers = localhost:9093
A->B.enabled = true
A->B.topics = .*
B->A.enabled = false
checkpoints.topic.replication.factor=1
heartbeats.topic.replication.factor=1
offset-syncs.topic.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1
config.storage.replication.factor=1
Unfortunately MirrorMaker 2 currently does not fully support multiple instances when running in dedicated cluster mode (connect-mirror-maker.sh). This is going to be fixed with KIP-710.
As long as this is not fixed I would not recommend to run multiple instances of MirrorMaker in dedicated cluster mode, because of the failure scenarios described in the KIP.
2 Likes