Mirrormaker2 creating heartbeats topics recursively in loop in active-active mode

Kafka Connect
1

We are experiencing an issue with Apache Kafka MirrorMaker 2 (MM2) where it is creating heartbeat topics in a recursive manner. This results of heartbeat topics, which is not the intended behavior. This issue specifically arises when we enable reverse replication; one-way replication works fine.

Issue Observed:

  • This recursive creation of heartbeat topics is leading to an excessive number of topics being generated, which is not manageable.
  • While replicating from a source with version 2.8.0 to a destination with version 3.4.1, everything works fine. However, when performing reverse replication from a destination with version 3.4.1 to a source with version 2.8.0, it creates heartbeat topics recursively.

Environment Details:

  • Source Apache Kafka Version: 2.8.0

  • Destination Apache Kafka Version: 3.4.1

MM2 properties being used on source and destination cluster:

## Mirror Maker Configurations

# name of the connector, e.g. "us-west->us-east"

name = {{ source_cluster_name }}-{{ dest_cluster_name }}

# Maximum number of tasks to use for this connector

tasks.max = 4

# Setting replication factor of newly created remote topics

replication.factor = 3

errors.log.enable = true

errors.log.include.messages = true

# use ByteArrayConverter to ensure that records are not re-encoded and stay the same

key.converter = org.apache.kafka.connect.converters.ByteArrayConverter

value.converter = org.apache.kafka.connect.converters.ByteArrayConverter

## Kafka clusters aliases

clusters = {{ source_cluster_name }}, {{ dest_cluster_name }}

# upstream cluster to replicate

{{ source_cluster_name }}.bootstrap.servers = {{ source_cluster_ips }}

# downstream cluster

{{ dest_cluster_name }}.bootstrap.servers = {{ dest_cluster_ips }}

# enable and configure individual replication flows

{{ source_cluster_name }}->{{ dest_cluster_name }}.enabled = true

# whether or not to monitor source cluster for configuration changes

{{ source_cluster_name }}->{{ dest_cluster_name }}.sync.topics.configs.enabled = true

# regex of topics to replicate, e.g. "topic1|topic2|topic3". Comma-separated lists are also supported.

{{ source_cluster_name }}->{{ dest_cluster_name }}.topics = {{ topics_to_replicate }}

# Configure from when the MM2 shuld start replicating data

{{ source_cluster_name }}->{{ dest_cluster_name }}.consumer.auto.offset.reset = latest

# Sync consumer group offsets

{{ source_cluster_name }}->{{ dest_cluster_name }}.exclude.internal.topics = false

{{ source_cluster_name }}->{{ dest_cluster_name }}.emit.heartbeats.enabled = true

# Enable automated consumer offset sync

{{ source_cluster_name }}->{{ dest_cluster_name }}.sync.group.offsets.enabled = true

{{ source_cluster_name }}->{{ dest_cluster_name }}.emit.checkpoints.enabled = true

offset.flush.timeout.ms = 60000

We appreciate your assistance in resolving this critical issue.

2

hey @Ismail

welcome :slight_smile:

is my understanding correct that you end with something as described here
https://issues.apache.org/jira/browse/KAFKA-9914

best,
michael

3

Hey @mmuehlbeyer

Many thanks for your response.

You got it correctly, we are facing similar issue as mentioned in [KAFKA-9914].

One way replication is working fine [ A → B ]. We are seeing recursive heartbeats topics when we enable reverse replication. [ A ← B ]. We even tried with different unique cluster names for source and destinations but ended up with the same issue.

  • A Kafka Version: 2.8.0
  • B Kafka Version: 3.4.1

We appreciate your valuable inputs.

Thanks again

1 Like
4

hey @Ismail

very welcome
afaik cycle detection should prevent the infinite creation of topics as described here
https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP382:MirrorMaker2.0-InternalTopics

5

@mmuehlbeyer

I reviewed the Confluence document you provided, but I’m still unsure about the steps. Could you please help us to provide more details on how we should proceed?

6

basically I think this should be handled automatically

you’re running MM2, right?

7

yes, we are running mm2

8

mirrormaker standalone or kafka connect? just to be sure, will try to rebuild

9

We are using kafka connect(distributed process across 3 nodes). And we are using exact same mm2.properties as mentioned above in topic

10

thx might take some time
will let you know

1 Like
11

Thanks for your help. I appreciate it.

12

Hi @mmuehlbeyer

Hope you’re doing well. I just wanted to follow up on this issue. Whenever you have an update, please let us know.

Thank You

13

might take some time till checked

will keep you posted

14

Thanks for the response.

15

just testing the setup
would be so kind and shared the exact config (without passwords etc)?

16

We are using the properties mentioned above in the link, and they are included in the question we asked earlier.

17

thanks will take a look

18

Sure, Thank you
Will wait for your response

19

was not able to get it running either
let me check something else

20

Many thanks for the update.
Sure, no worries.
Will wait for your response