MirrorMaker 2.0 multiple instances

Ops Cluster Replication
1

Hi

I have two sites (s1, s2) with kafka. On each site i have “topic-1” and “topic-2”. It means s1 has topic-1 and topic-2 and s2 has topic-1 and topic-2. 4 topics in total.
I want to aggregate all messages from all “topic-1” topics on s1 and all messages from all “topic-2” topics on s2

For this purpose i am using MirrorMaker 2.0: connect-mirror-maker (that is using org.apache.kafka.connect.mirror.MirrorMaker)

As soon as it is recommended to have MirrorMaker (mm) closer to target topic (producer) i am trying to configure two instances of mm
for each instance i have separate properties file

  • on s1 i have s2->s1.topics=topic-1
  • on s2 i have s1->s2.topics=topic-2

additionally i override default replication policy to not create additional topics on target servers. It means that all messages from s1 “topic2” should be replicated to existing s2 “topic-2” instead of creating “s1.topic-2” there

replication.policy.class=com.my_project.mm2.PrefixlessReplicationPolicy

When i started mm on site 1 it was working fine: all messages from s2.topic-1 was moved to s1.topic-1 in addition to existing messages.
But i saw strange messages in logs:

[org.apache.kafka.connect.mirror.MirrorMaker:lambda$configureConnector$3:212] Connector s1->s2 configured.
[org.apache.kafka.connect.mirror.MirrorMaker:lambda$configureConnector$3:212] Connector s2->s1 configured.

It is strange for me because i need only s2->s1 connector on site 1. I dont have configuration for s1->s2 there
Then i started mm on s2 and it didn’t worked as i expected: messages from s1.topic-2 wasnt moved to s2.topic-2. This is log for second instance of mm

[org.apache.kafka.connect.mirror.MirrorMaker:lambda$configureConnector$3:210] Connector s2->s1 is a follower. Using existing configuration.
[org.apache.kafka.connect.mirror.MirrorMaker:lambda$configureConnector$3:210] Connector s2->s1 is a follower. Using existing configuration.

Please help me understand how to solve this problem.
As i understand first instance of mm created 2 connectors

  • one for s1->s2 that is doing nothing (i checked: messages from s1.topic-1 isn’t replicated to s2.topic-2)
  • one for s2->s1 that is doing what i want

Then second instance of mm is starting. It sees that “s1->s2” and “s2->s1” connectors already presented (where? this second mm instance started on different site) and just reusing existing config instead of its own. As result i am getting two nodes of s2->s1.topics=topic-1 mm
Am i correct? Why first instance of mm created “s1->s2” connector?

I tried to rename clusters in config for second mm so it became s2-1->s2-2.topics=topic-2
after that it started to work as i expected: messages from s1.topic-2 was moved to s2.topic-2
Is it correct approach? should i have unique names for clusters on each mm instance?

p.s. i am using confluent 6.1.0

2

about com.my_project.mm2.PrefixlessReplicationPolicy it is very simple:

public class PrefixlessReplicationPolicy extends DefaultReplicationPolicy {

@Override
public String formatRemoteTopic(String sourceClusterAlias, String topic) {
    return topic;
}

}

3

Hi

can anybody help me with that?

4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.