I’m wondering if it’s possible to run multiple instances of MM2 per data center? I am trying to run 2 instances of connect-mirror-maker.sh per data center for scalability purposes. However, I cannot get 2 instances to work at the same time: once I start up the second mirror maker, it takes over for the first one, and the first one completely stops replicating.
What am I missing here? Would greatly appreciate any guidance.
welcome to the forum
could you elaborate a bit more?
what would you like to achieve ? replicate the same topic with two independent mirror maker instances?
Hi @mmuehlbeyer, thanks for your response
I would like to replicate Kafka events between two data centers with two independent mirror maker instances, in a way that lets me load balance between the two mirror makers rather than have the two instances both do the same work. So, if I have my mirror makers replicating from A to B, I would like one instance to handle part of the produced events and the other instance to handle the rest. And if one mirror maker instance were to go down, the process would rebalance so that the remaining instance would pick up from where it left off and become responsible for all replication until the second instance becomes available again. That could mean that both mirror makers are replicating different parts of the same topic, but NOT that they are both replicating the whole topic
Thank you so much
basically it should be possible.
did you configure as the documentation states ?
is there some traffic going on the topics?
Yes, I’m using below configuration, and there is traffic in the topic.
clusters = A, B A.bootstrap.servers = localhost:9092 B.bootstrap.servers = localhost:9093 A->B.enabled = true A->B.topics = .* B->A.enabled = false checkpoints.topic.replication.factor=1 heartbeats.topic.replication.factor=1 offset-syncs.topic.replication.factor=1 offset.storage.replication.factor=1 status.storage.replication.factor=1 config.storage.replication.factor=1
Unfortunately MirrorMaker 2 currently does not fully support multiple instances when running in dedicated cluster mode (
connect-mirror-maker.sh). This is going to be fixed with KIP-710.
As long as this is not fixed I would not recommend to run multiple instances of MirrorMaker in dedicated cluster mode, because of the failure scenarios described in the KIP.