I am trying to enable replication between two different Kafka clusters (cluster A and cluster B). I do not require active/active replication. Going from A → B is enough for me.
To achieve that goal I plan to set up a Mirror Maker 2 cluster such as data flows as follows:
cluster A → Mirror Maker 2 cluster → cluster B
Cluster A has close to 3k partitions and it is part of a business-critical application so I have to guarantee speedy replication between A and B.
I started experimenting with setting up a dedicated mirror maker cluster (i.e. using
KIP-382 mentions that “In this mode, MirrorMaker does not require an existing Connect cluster. Instead, a high-level driver manages a collection of Connect workers.”. However, I am struggling with how to make multiple nodes communicate amongst themselves to divide up the load.
I have tried multiple configs such as
rest.advertised.port, and others.
None of them seem to enable communication between MM2 workers. Can someone point me to an example configuration on how to achieve that?
Edited: provided more clarity around my use case.