How to setup a dedicated mirror maker 2 cluster with multiple nodes

Kafka Connect
1

Hi,

I am trying to enable replication between two different Kafka clusters (cluster A and cluster B). I do not require active/active replication. Going from A → B is enough for me.

To achieve that goal I plan to set up a Mirror Maker 2 cluster such as data flows as follows:

cluster A → Mirror Maker 2 cluster → cluster B

Cluster A has close to 3k partitions and it is part of a business-critical application so I have to guarantee speedy replication between A and B.

I started experimenting with setting up a dedicated mirror maker cluster (i.e. using connect-mirror-maker.sh).

KIP-382 mentions that “In this mode, MirrorMaker does not require an existing Connect cluster. Instead, a high-level driver manages a collection of Connect workers.”. However, I am struggling with how to make multiple nodes communicate amongst themselves to divide up the load.

I have tried multiple configs such as listeners, rest.advertised.host.name, rest.advertised.port, and others.

None of them seem to enable communication between MM2 workers. Can someone point me to an example configuration on how to achieve that?

Edited: provided more clarity around my use case.

2

Hi @maurolscla

could you provide some details what you would like to achieve?
It’s not quite clear to me.

I assume you have a kafka cluster running and would like to mirror it with mirrormaker2, correct?

best,
michael

3

Hi @mmuehlbeyer, thanks for your reply. I have edited my OP to provide some clarity around my use case. Is that clear now?

4

Hi @maurolscla

thanks for the update.
understood your setup.

tried it locally and faced the same issue as you did, will dig a little deeper.

best,
michael

5

mmh I think it’s not possible

out of the linked KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters - Apache Kafka - Apache Software Foundation

Since REST is not available in dedicated MM2 mode, the notification never reaches the leader, rendering the cluster unable to detect config changes.

6

@mmuehlbeyer good catch!
Thanks for pointing me in the right direction. It seems that I’ll have to resort to Connect mode to get this going.
It’s unfortunate that documentation is not clear about this.

1 Like
7

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.