MirrorMaker2 not replicating certain topics

alexei.ivanov · 14 April 2025 10:35

Hello everyone!

I have installed a MirrorMaker2 cluster with 2 nodes that is replicating a 3 broker + zookeeper Kafka cluster with just under 1100 topics. The replication works as expected for all but the following topics:

STAGE.OU.task-updated-ics2-internal
STAGE.OU.task-updated-internal
STAGE.OU.task-upsert-events-internal
STAGE.OU.task-upsert-ics2-events-internal
TEST.OU.task-updated-ics2-internal
TEST.OU.task-updated-internal
TEST.OU.task-upsert-events-internal
TEST.OU.task-upsert-ics2-events-internal

There are several equivalent topics for a different environment that are replicated, however that is not important since they have no messages.

I do not know if it is of any relevance but the message count is as follows:

STAGE.OU.task-updated-ics2-internal - 55873
STAGE.OU.task-updated-internal - 420987
STAGE.OU.task-upsert-events-internal - 15000000+
STAGE.OU.task-upsert-ics2-events-internal - 43502
TEST.OU.task-updated-ics2-internal - 0
TEST.OU.task-updated-internal - 35
TEST.OU.task-upsert-events-internal - 14461
TEST.OU.task-upsert-ics2-events-internal - 0

I have made sure to include all topics through the mm2.properties configuration and that no topics are being excluded:

topics=.*
topics.exclude=

I have also tried replicating these topics alone without success:

topics=STAGE.OU.task-updated-ics2-internal,STAGE.OU.task-updated-internal,STAGE.OU.task-upsert-events-internal,STAGE.OU.task-upsert-ics2-events-internal,TEST.OU.task-updated-ics2-internal,TEST.OU.task-updated-internal,TEST.OU.task-upsert-events-internal,TEST.OU.task-upsert-ics2-events-internal

I am using the IdentityReplicationPolicy replication policy class. There is no indication of an error in the MirrorMaker log.

Source cluster is at version 7.6.5-ccs
Target cluster is at version 7.6.4-ccs
MirrorMaker cluster is at version 7.6.5-ccs

Any idea why these topics are excluded from replication?

mmuehlbeyer · 14 April 2025 11:40

hey @alexei.ivanov

I’m not fully sure only thinking out loud:

may this be related to the topic naming?
as all the topic have internal in their names…

there is some logic around the internal topics in MM2 so maybe it’s somehow related to this

alexei.ivanov · 15 April 2025 10:35

Hi @mmuehlbeyer! Thanks for a quick response.

may this be related to the topic naming? as all the topic have internal in their names… there is some logic around the internal topics in MM2 so maybe it’s somehow related to this

That was my initial thought as well. However, I do see that topics that have the same name but different environment prefix (also end with internal in their names) are being replicated just fine. So, I feel the theory is somewhat good, but not 100% accurate. Perhaps there is a glitch in this logic surrounding internal topics? Also, I have tried configuring exclude.internal.topics and setting it to false, but MirrorMaker seems to ignore this configuration no matter which way it is configured. Any other ideas/insights will be greatly appreciated!

mmuehlbeyer · 16 April 2025 11:22

hmm pretty strange need
to dig a bit around.

maybe something worth to try
what about a second MM2 config/cluster which only serves the topics not getting
replicated at the moment?
maybe we can gain some further insights around the behaviour with setting everything to debug then?
(I’ve seen you already tried but probably within the same config/cluster?)

mmuehlbeyer · 16 April 2025 11:33

another one :
what happens if you switch to the default replication policy?

I’ve just checked

github.com/apache/kafka

connect/mirror-client/src/main/java/org/apache/kafka/connect/mirror/ReplicationPolicy.java

58ad29cbf


      
           * Returns the name of heartbeats topic.
           */
          default String heartbeatsTopic() {
              return "heartbeats";
          }
          
          /**
           * Returns the name of the offset-syncs topic for given cluster alias.
           */
          default String offsetSyncsTopic(String clusterAlias) {
              return "mm2-offset-syncs." + clusterAlias + ".internal";
          }
          
          /**
           * Returns the name of the checkpoints topic for given cluster alias.
           */
          default String checkpointsTopic(String clusterAlias) {
              return clusterAlias + ".checkpoints.internal";
          }
          
          /**

according to that (line 97 following) you should not be hit by the internal topic check

default boolean isMM2InternalTopic(String topic) {
        return  topic.startsWith("mm2") && topic.endsWith(".internal") || isCheckpointsTopic(topic);
    }

alexei.ivanov · 16 April 2025 15:40

Thanks @mmuehlbeyer!

After looking at the code a bit closer I may have discovered the issue.

according to that (line 97 following) you should not be hit by the internal topic check

The code you referred to is a bit newer than the one I am currently running. The version was updated December 2024, so we need to update our clusters to version 7.8 or 7.9 (currently running 7.6.5). Take a look at this updated segment:

https://github.com/apache/kafka/commit/0bbed823e818a920fbaa2d54b50f4fcd81a8a759#diff-20a9931da3de4b577145af409d7e704e90ed30b410e4d8236e297b033dea6d0aR100-L109

However, it still puzzles me how 2 of my 10 topics that end with “-internal” in their name got replicated. Anyway, I think I need to update the clusters before continuing finding out what is wrong.

Thanks for pointing me in the right direction!
Btw, DEBUG/TRACE log mode did not produce any leads.

P.S. I am off on easter break so it might take a while before I can back to you with any results.

alexei.ivanov · 23 April 2025 12:55

Hello again,

Just updated the cluster to version 7.9.0 using but to no avail. It seem for some reason the code is not updated. I was unable to verify this by inspecting DefaultReplicationPolicy class, but the DefaultTopicFilter class still has the old topic filter as topics.exclude was still set to [ .*[\-\.]internal, .*\.replica, __.* ] by default. This raises the obvious question: in what version of Confluent will this feature be available?

EDIT: KAFKA-17200 is part of Kafka version 4.0, so the question is actually about what version of Confluent will include Kafka 4.0? Thanks!

mmuehlbeyer · 27 April 2025 20:48

Hi @alexei.ivanov

I see thanks for providing the insights.
as said one possible workaround might be to provide a custom replication policy.
(or adapt the code and compile yourself, probably not cool if you need support or would like to use Confluent platform)

Confluent plaform 8 will be built on Apache Kafka 4

in preparation for the next Confluent Platform release built on Apache Kafka 4.0
[…]
Confluent Platform 8.0, coming in Summer 2025

for ref

alexei.ivanov · 2 May 2025 12:24

Hi, @mmuehlbeyer!
Thanks, although I was able to discover that page on my own I wanted to see if a more specific release date could be provided.

We have considered the workarounds (code adaptation and custom policy) and came to the conclusion that they are not sustainable and are therefore too risky to implement. We ultimately decided to wait for the new version of Confluent to be released. But it is comforting to know that the workarounds we thought of were the same.

Thanks again for your help!

system · 9 May 2025 12:25

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Unable to mirror few topics with source 2.1 and destination 3.7 Cluster Replication	0	12	13 February 2025
MirrorMaker 2 without internal topics Cluster Replication	2	6406	14 December 2022
MirrorMaker 2: topic gets created on target but records stay (or loop) on source Cluster Replication	0	14	24 April 2025
MirrorMaker 2.0 multiple instances Cluster Replication	3	6227	20 June 2021
Replicator creating topics but not migrating topic messages Cluster Replication	15	1521	18 December 2023

MirrorMaker2 not replicating certain topics

Related topics