Mirrormaker2 creating heartbeats topics recursively in loop in active-active mode

Kafka Connect
50

Bidirectional load test was successful.

We used below config file:

## Mirror Maker Configurations
# name of the connector, e.g. "us-west->us-east"
name = A-B

# Maximum number of tasks to use for this connector
tasks.max = 3

# Setting replication factor of newly created remote topics
replication.factor = 3

errors.log.enable = true
errors.log.include.messages = true

# use ByteArrayConverter to ensure that records are not re-encoded and stay the same
key.converter = org.apache.kafka.connect.converters.ByteArrayConverter
value.converter = org.apache.kafka.connect.converters.ByteArrayConverter

## Kafka clusters aliases
clusters = A, B


# upstream cluster to replicate
A.bootstrap.servers = x.x.x.x:9092,y.y.y.y:9092,z.z.z.z:9092


# downstream cluster
B.bootstrap.servers = x.x.x.x:9092,y.y.y.y:9092,z.z.z.z:9092

# enable and configure individual replication flows
A->B.enabled = true
B->A.enabled = true

# whether or not to monitor source cluster for configuration changes
A->B.sync.topics.configs.enabled = true
B->A.sync.topics.configs.enabled = true

# regex of topics to replicate, e.g. "topic1|topic2|topic3". Comma-separated lists are also supported.
A->B.topics =  xxxxx|yyyyy|zzzzz
B->A.topics =  aaaaa-2.8|bbbbb-2.8|ccccc-2.8

# Configure from when the MM2 shuld start replicating data
A->B.consumer.auto.offset.reset = latest
B->A.consumer.auto.offset.reset = latest

# Sync consumer group offsets
A->B.exclude.internal.topics = false
A->B.emit.heartbeats.enabled = true

B->A.exclude.internal.topics = false
B->A.emit.heartbeats.enabled = true

# Enable automated consumer offset sync
A->B.sync.group.offsets.enabled = true
A->B.emit.checkpoints.enabled = true

B->A.sync.group.offsets.enabled = true
B->A.emit.checkpoints.enabled = true

offset.flush.timeout.ms = 60000


# SASL Configurations

B.security.protocol=SASL_PLAINTEXT
B.sasl.mechanism=SCRAM-SHA-256
B.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="aaaaaa";

A.security.protocol=SASL_PLAINTEXT
A.sasl.mechanism=SCRAM-SHA-256
A.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="bbbbbb";

Key findings:

I discovered that MirrorMaker 2 (MM2) will replicate bidirectionally if it is running on the higher Kafka version between two clusters. For example, we have a 2.8.0 cluster and a 3.4.1 cluster. Previously, active/active replication was not working as expected on the 2.8.0 cluster. However, when we tried running MM2 on the 3.4.1 cluster, it worked well after restarting MM2 a few times (initially, 2.8.0 —> 3.4.1 replication was not working, but it started working after a few restarts).

Additionally, setting tasks.max = 3 is important if you plan to run the MirrorMaker process on 3 Connect nodes. Previously, we were using a value of 4, and it was not transmitting the data—only creating the prefix topics.

Ref link-

1 Like