Mirrormaker2 creating heartbeats topics recursively in loop in active-active mode

mmuehlbeyer · 11 July 2024 19:28

mmh need to check in detail.

just to be sure from which Kafka Home did you start mirrormaker?

Ismail · 12 July 2024 10:39

Our service file is as follows:

cat /usr/lib/systemd/system/mm2.service
[Unit]
Description=MirrorMaker2
ConditionPathExists=/code/kafka/3.4.1/mm/mm2/mm2.properties
After=network.target

[Service]
Type=simple
#User=kafka
#SyslogIdentifier=kafka-mm2
EnvironmentFile=/code/kafka/3.4.1/mm/heap_env_file.txt
Environment="KAFKA_JMX_OPTS=-javaagent:/code/kafka/3.4.1/libs/jmx_prometheus_javaagent-0.17.0.jar=9191:/code/kafka/3.4.1/config/kafka_connect.yml"
ExecStart=/code/kafka/3.4.1/bin/connect-mirror-maker.sh /code/kafka/3.4.1/mm/mm2/mm2.properties
ExecStop=/usr/bin/pkill /code/kafka/3.4.1/bin/connect-mirror-maker.sh
#SuccessExitStatus=143
#Restart=always
#RestartSec=5
#StartLimitInterval=0
TimeoutSec=600s


[Install]
WantedBy=multi-user.target

Ismail · 12 July 2024 11:58

@mmuehlbeyer We tried to run MM2 on kafka cluster 3.4.1, and we pass bidirectional replication as mentioned in below properties. Mirrormaker is able to create the topics both the side, and we are not seeing any recursive heartbeat topics(which is good progress as of now). But the weird thing is happening is, it’s not replicating data to the topics in both ways, we tested manually on the console producer on 3.4.1 and console consumer on 2.8.0 and vice versa.

NOTE- This is single connect MM2 process which will do bidirectional replication

## Mirror Maker Configurations
# name of the connector, e.g. "us-west->us-east"
name = A-B

# Maximum number of tasks to use for this connector
tasks.max = 4

# Setting replication factor of newly created remote topics
replication.factor = 3

errors.log.enable = true
errors.log.include.messages = true

# use ByteArrayConverter to ensure that records are not re-encoded and stay the same
key.converter = org.apache.kafka.connect.converters.ByteArrayConverter
value.converter = org.apache.kafka.connect.converters.ByteArrayConverter

## Kafka clusters aliases
clusters = A, B


# upstream cluster to replicate
A.bootstrap.servers = x.x.x.x:9092,y.y.y.y:9092,z.z.z.z:9092


# downstream cluster
B.bootstrap.servers = x.x.x.x:9092,y.y.y.y:9092,z.z.z.z:9092

# enable and configure individual replication flows
A->B.enabled = true
B->A.enabled = true

# whether or not to monitor source cluster for configuration changes
A->B.sync.topics.configs.enabled = true
B->A.sync.topics.configs.enabled = true

# regex of topics to replicate, e.g. "topic1|topic2|topic3". Comma-separated lists are also supported.
A->B.topics =  topic1|topic2|topic3
B->A.topics =  aaaa-2.8|bbbb-2.8|cccc-2.8

# Configure from when the MM2 shuld start replicating data
A->B.consumer.auto.offset.reset = latest
B->A.consumer.auto.offset.reset = latest

# Sync consumer group offsets
A->B.exclude.internal.topics = false
A->B.emit.heartbeats.enabled = true

B->A.exclude.internal.topics = false
B->A.emit.heartbeats.enabled = true

# Enable automated consumer offset sync
A->B.sync.group.offsets.enabled = true
A->B.emit.checkpoints.enabled = true

B->A.sync.group.offsets.enabled = true
B->A.emit.checkpoints.enabled = true

offset.flush.timeout.ms = 60000


# SASL Configurations


B.security.protocol=SASL_PLAINTEXT
B.sasl.mechanism=SCRAM-SHA-256
B.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="xxxxx";

A.security.protocol=SASL_PLAINTEXT
A.sasl.mechanism=SCRAM-SHA-256
A.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="yyyyy";

Will single MM2 process handle 2 way replications??

mmuehlbeyer · 14 July 2024 11:55

Will single MM2 process handle 2 way replications??

afaik it should what happens if you switch to one way only?

Ismail · 15 July 2024 09:03

Quick update:

I tried single MM2 process that will handle 2 way replications, and set the param tasks.max = 3 now 3.4.1 to 2.8.0 working

but now stuck with 2.8.0 to 3.4.1 - it’s creating the prefix topics but its not replicating data

Note: I am running MM2 connect process on 3.4.1 cluster for bidirectional replication

Ismail · 15 July 2024 09:11

Regarding your question:

what happens if you switch to one way only?

We are experiencing issues with MM2 when attempting one-way replication between:

When running MM2 on source Kafka version 3.4.1 to send data to destination Kafka version 2.8.0, the prefix topics are created, but data is not being transmitted.
When running MM2 on source Kafka version 2.8.0 to send data to destination Kafka version 3.4.1, the prefix topics are created, but data is not being transmitted.
When running MM2 on destination Kafka version 3.4.1 to fetch data from source Kafka version 2.8.0, it works perfectly fine, creating the prefix topics and transmitting data successfully.
When running MM2 on destination Kafka version 2.8.0 to fetch data from source Kafka version 3.4.1, it does not even create the prefix topics.

mmuehlbeyer · 15 July 2024 11:14

pretty interesting and confusing

let me dig a bit into that probably tomorrow.

Ismail · 15 July 2024 11:33

quick update:

bidirectional is working from 3.4.1 ----2.8.0 and 2.8.0----->3.4.1

we are doing load test and update you

mmuehlbeyer · 15 July 2024 12:09

cool

I’m very interested in the final config

Ismail · 15 July 2024 12:57

Bidirectional load test was successful.

We used below config file:

## Mirror Maker Configurations
# name of the connector, e.g. "us-west->us-east"
name = A-B

# Maximum number of tasks to use for this connector
tasks.max = 3

# Setting replication factor of newly created remote topics
replication.factor = 3

errors.log.enable = true
errors.log.include.messages = true

# use ByteArrayConverter to ensure that records are not re-encoded and stay the same
key.converter = org.apache.kafka.connect.converters.ByteArrayConverter
value.converter = org.apache.kafka.connect.converters.ByteArrayConverter

## Kafka clusters aliases
clusters = A, B


# upstream cluster to replicate
A.bootstrap.servers = x.x.x.x:9092,y.y.y.y:9092,z.z.z.z:9092


# downstream cluster
B.bootstrap.servers = x.x.x.x:9092,y.y.y.y:9092,z.z.z.z:9092

# enable and configure individual replication flows
A->B.enabled = true
B->A.enabled = true

# whether or not to monitor source cluster for configuration changes
A->B.sync.topics.configs.enabled = true
B->A.sync.topics.configs.enabled = true

# regex of topics to replicate, e.g. "topic1|topic2|topic3". Comma-separated lists are also supported.
A->B.topics =  xxxxx|yyyyy|zzzzz
B->A.topics =  aaaaa-2.8|bbbbb-2.8|ccccc-2.8

# Configure from when the MM2 shuld start replicating data
A->B.consumer.auto.offset.reset = latest
B->A.consumer.auto.offset.reset = latest

# Sync consumer group offsets
A->B.exclude.internal.topics = false
A->B.emit.heartbeats.enabled = true

B->A.exclude.internal.topics = false
B->A.emit.heartbeats.enabled = true

# Enable automated consumer offset sync
A->B.sync.group.offsets.enabled = true
A->B.emit.checkpoints.enabled = true

B->A.sync.group.offsets.enabled = true
B->A.emit.checkpoints.enabled = true

offset.flush.timeout.ms = 60000


# SASL Configurations

B.security.protocol=SASL_PLAINTEXT
B.sasl.mechanism=SCRAM-SHA-256
B.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="aaaaaa";

A.security.protocol=SASL_PLAINTEXT
A.sasl.mechanism=SCRAM-SHA-256
A.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="bbbbbb";

Key findings:

I discovered that MirrorMaker 2 (MM2) will replicate bidirectionally if it is running on the higher Kafka version between two clusters. For example, we have a 2.8.0 cluster and a 3.4.1 cluster. Previously, active/active replication was not working as expected on the 2.8.0 cluster. However, when we tried running MM2 on the 3.4.1 cluster, it worked well after restarting MM2 a few times (initially, 2.8.0 —> 3.4.1 replication was not working, but it started working after a few restarts).

Additionally, setting tasks.max = 3 is important if you plan to run the MirrorMaker process on 3 Connect nodes. Previously, we were using a value of 4, and it was not transmitting the data—only creating the prefix topics.

Ref link-

mmuehlbeyer · 15 July 2024 19:01

thanks for letting us know

so I assume everything works as expected?

Ismail · 16 July 2024 09:06

Yes, it is working as expected. As we had tested the actual required workflow as:

Workflow:
A--->B
B--->A

C--->B
B--->C

Versions:
A = 2.8.0
B = 3.4.1
C = 2.5.0

All this workflows are running in MM2 connect process on B cluster (3.4.1 version)

Once again, Many thanks for your help here

mmuehlbeyer · 16 July 2024 11:25

good to hear
always a pleasure

system · 23 July 2024 11:26

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Kafka Mirror Maker 2 - source topic keep getting created Kafka Streams	2	3683	11 August 2021
MirrorMaker 2 without internal topics Cluster Replication	2	5632	14 December 2022
MirrorMaker2 re creating deleted topics Kafka Connect	6	2651	8 April 2023
I've a massive amount of lag when i try to consume MM2 replicated topics Kafka Connect	3	2767	10 February 2023
MirrorMaker Heartbeats Keep Getting Replicated Cluster Replication	0	14	15 October 2024

Mirrormaker2 creating heartbeats topics recursively in loop in active-active mode

Related topics