# Mirrormaker2 creating heartbeats topics recursively in loop in active-active mode

**URL:** <https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767>\
**Category:** Kafka Connect\
**Created:** [24 May 2024 18:47 UTC](https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767 "2024-05-24T18:47:27Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ismail](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/ismail/32/3801_2.png) [@Ismail](https://forum.confluent.io/u/Ismail)\
**Post date:** [24 May 2024 18:47 UTC](https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767/1 "2024-05-24T18:47:27Z")

</div>

We are experiencing an issue with Apache Kafka MirrorMaker 2 (MM2) where it is creating heartbeat topics in a recursive manner. This results of heartbeat topics, which is not the intended behavior. This issue specifically arises when we enable reverse replication; one-way replication works fine.

#### Issue Observed:

- This recursive creation of heartbeat topics is leading to an excessive number of topics being generated, which is not manageable.
- While replicating from a source with version 2.8.0 to a destination with version 3.4.1, everything works fine. However, when performing reverse replication from a destination with version 3.4.1 to a source with version 2.8.0, it creates heartbeat topics recursively.

#### Environment Details:

- Source Apache Kafka Version: 2.8.0

- Destination Apache Kafka Version: 3.4.1

#### MM2 properties being used on source and destination cluster:

```auto
## Mirror Maker Configurations

# name of the connector, e.g. "us-west->us-east"

name = {{ source_cluster_name }}-{{ dest_cluster_name }}

# Maximum number of tasks to use for this connector

tasks.max = 4

# Setting replication factor of newly created remote topics

replication.factor = 3

errors.log.enable = true

errors.log.include.messages = true

# use ByteArrayConverter to ensure that records are not re-encoded and stay the same

key.converter = org.apache.kafka.connect.converters.ByteArrayConverter

value.converter = org.apache.kafka.connect.converters.ByteArrayConverter

## Kafka clusters aliases

clusters = {{ source_cluster_name }}, {{ dest_cluster_name }}

# upstream cluster to replicate

{{ source_cluster_name }}.bootstrap.servers = {{ source_cluster_ips }}

# downstream cluster

{{ dest_cluster_name }}.bootstrap.servers = {{ dest_cluster_ips }}

# enable and configure individual replication flows

{{ source_cluster_name }}->{{ dest_cluster_name }}.enabled = true

# whether or not to monitor source cluster for configuration changes

{{ source_cluster_name }}->{{ dest_cluster_name }}.sync.topics.configs.enabled = true

# regex of topics to replicate, e.g. "topic1|topic2|topic3". Comma-separated lists are also supported.

{{ source_cluster_name }}->{{ dest_cluster_name }}.topics = {{ topics_to_replicate }}

# Configure from when the MM2 shuld start replicating data

{{ source_cluster_name }}->{{ dest_cluster_name }}.consumer.auto.offset.reset = latest

# Sync consumer group offsets

{{ source_cluster_name }}->{{ dest_cluster_name }}.exclude.internal.topics = false

{{ source_cluster_name }}->{{ dest_cluster_name }}.emit.heartbeats.enabled = true

# Enable automated consumer offset sync

{{ source_cluster_name }}->{{ dest_cluster_name }}.sync.group.offsets.enabled = true

{{ source_cluster_name }}->{{ dest_cluster_name }}.emit.checkpoints.enabled = true

offset.flush.timeout.ms = 60000

```

We appreciate your assistance in resolving this critical issue.

---

<div class="post-metadata">

**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\
**Post date:** [29 May 2024 08:07 UTC](https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767/2 "2024-05-29T08:07:53Z")

</div>

hey @Ismail

welcome 🙂

is my understanding correct that you end with something as described here  
[https://issues.apache.org/jira/browse/KAFKA-9914](https://issues.apache.org/jira/browse/KAFKA-9914)

best,  
michael

---

<div class="post-metadata">

**Author:** ![Ismail](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/ismail/32/3801_2.png) [@Ismail](https://forum.confluent.io/u/Ismail)\
**Post date:** [29 May 2024 10:26 UTC](https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767/3 "2024-05-29T10:26:09Z")

</div>

Hey @mmuehlbeyer

Many thanks for your response.

You got it correctly, we are facing similar issue as mentioned in [KAFKA-9914].

One way replication is working fine [A → B]. We are seeing recursive heartbeats topics when we enable reverse replication. [A ← B]. We even tried with different unique cluster names for source and destinations but ended up with the same issue.

- A Kafka Version: 2.8.0
- B Kafka Version: 3.4.1

We appreciate your valuable inputs.

Thanks again

---

<div class="post-metadata">

**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\
**Post date:** [29 May 2024 12:30 UTC](https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767/4 "2024-05-29T12:30:27Z")

</div>

hey @Ismail

very welcome  
afaik cycle detection should prevent the infinite creation of topics as described here  
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP382:MirrorMaker2.0-InternalTopics](https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP382:MirrorMaker2.0-InternalTopics)

---

<div class="post-metadata">

**Author:** ![Ismail](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/ismail/32/3801_2.png) [@Ismail](https://forum.confluent.io/u/Ismail)\
**Post date:** [29 May 2024 13:00 UTC](https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767/5 "2024-05-29T13:00:32Z")

</div>

@mmuehlbeyer

I reviewed the Confluence document you provided, but I’m still unsure about the steps. Could you please help us to provide more details on how we should proceed?

---

<div class="post-metadata">

**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\
**Post date:** [29 May 2024 14:11 UTC](https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767/6 "2024-05-29T14:11:48Z")

</div>

basically I think this should be handled automatically

you’re running MM2, right?

---

<div class="post-metadata">

**Author:** ![Ismail](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/ismail/32/3801_2.png) [@Ismail](https://forum.confluent.io/u/Ismail)\
**Post date:** [29 May 2024 14:13 UTC](https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767/7 "2024-05-29T14:13:37Z")

</div>

yes, we are running mm2

---

<div class="post-metadata">

**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\
**Post date:** [30 May 2024 10:50 UTC](https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767/8 "2024-05-30T10:50:18Z")

</div>

mirrormaker standalone or kafka connect? just to be sure, will try to rebuild

---

<div class="post-metadata">

**Author:** ![Ismail](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/ismail/32/3801_2.png) [@Ismail](https://forum.confluent.io/u/Ismail)\
**Post date:** [30 May 2024 11:02 UTC](https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767/9 "2024-05-30T11:02:16Z")

</div>

We are using kafka connect(distributed process across 3 nodes). And we are using exact same mm2.properties as mentioned above in topic

---

<div class="post-metadata">

**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\
**Post date:** [30 May 2024 11:10 UTC](https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767/10 "2024-05-30T11:10:30Z")

</div>

thx might take some time  
will let you know

---

<div class="post-metadata">

**Author:** ![Ismail](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/ismail/32/3801_2.png) [@Ismail](https://forum.confluent.io/u/Ismail)\
**Post date:** [30 May 2024 11:21 UTC](https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767/11 "2024-05-30T11:21:27Z")

</div>

Thanks for your help. I appreciate it.

---

<div class="post-metadata">

**Author:** ![Ismail](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/ismail/32/3801_2.png) [@Ismail](https://forum.confluent.io/u/Ismail)\
**Post date:** [6 June 2024 10:38 UTC](https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767/12 "2024-06-06T10:38:39Z")

</div>

Hi @mmuehlbeyer

Hope you’re doing well. I just wanted to follow up on this issue. Whenever you have an update, please let us know.

Thank You

---

<div class="post-metadata">

**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\
**Post date:** [11 June 2024 09:01 UTC](https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767/13 "2024-06-11T09:01:07Z")

</div>

might take some time till checked

will keep you posted

---

<div class="post-metadata">

**Author:** ![Ismail](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/ismail/32/3801_2.png) [@Ismail](https://forum.confluent.io/u/Ismail)\
**Post date:** [12 June 2024 16:43 UTC](https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767/14 "2024-06-12T16:43:08Z")

</div>

Thanks for the response.

---

<div class="post-metadata">

**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\
**Post date:** [20 June 2024 06:49 UTC](https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767/15 "2024-06-20T06:49:59Z")

</div>

just testing the setup  
would be so kind and shared the exact config (without passwords etc)?

---

<div class="post-metadata">

**Author:** ![Ismail](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/ismail/32/3801_2.png) [@Ismail](https://forum.confluent.io/u/Ismail)\
**Post date:** [20 June 2024 10:15 UTC](https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767/16 "2024-06-20T10:15:13Z")

</div>

> [@Mirrormaker2 creating heartbeats topics recursively in loop in active-active mode](https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767#:~:text=MM2%20properties%20being%20used%20on%20source%20and%20destination%20cluster%3A):
>
> We are experiencing an issue with Apache Kafka MirrorMaker 2 (MM2) where it is creating heartbeat topics in a recursive manner. This results of heartbeat topics, which is not the intended behavior. This issue specifically arises when we enable reverse replication; one-way replication works fine. Issue Observed: This recursive creation of heartbeat topics is leading to an excessive number of topics being generated, which is not manageable. While replicating from a source with version 2.8.0 to a…

We are using the properties mentioned above in the link, and they are included in the question we asked earlier.

---

<div class="post-metadata">

**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\
**Post date:** [20 June 2024 11:19 UTC](https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767/17 "2024-06-20T11:19:29Z")

</div>

thanks will take a look

---

<div class="post-metadata">

**Author:** ![Ismail](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/ismail/32/3801_2.png) [@Ismail](https://forum.confluent.io/u/Ismail)\
**Post date:** [20 June 2024 11:30 UTC](https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767/18 "2024-06-20T11:30:09Z")

</div>

Sure, Thank you  
Will wait for your response

---

<div class="post-metadata">

**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\
**Post date:** [26 June 2024 08:29 UTC](https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767/19 "2024-06-26T08:29:19Z")

</div>

was not able to get it running either  
let me check something else

---

<div class="post-metadata">

**Author:** ![Ismail](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/ismail/32/3801_2.png) [@Ismail](https://forum.confluent.io/u/Ismail)\
**Post date:** [26 June 2024 12:29 UTC](https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767/20 "2024-06-26T12:29:10Z")

</div>

Many thanks for the update.  
Sure, no worries.  
Will wait for your response

[Next page](https://forum.confluent.io/t/mirrormaker2-creating-heartbeats-topics-recursively-in-loop-in-active-active-mode/10767.md?page=2)
