# MirrorMaker2 not replicating certain topics **URL:** \ **Category:** Cluster Replication\ **Created:** [14 April 2025 10:35 UTC](https://forum.confluent.io/t/mirrormaker2-not-replicating-certain-topics/37672 "2025-04-14T10:35:01Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![alexei.ivanov](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/alexei.ivanov/32/5049_2.png) [@alexei.ivanov](https://forum.confluent.io/u/alexei.ivanov)\ **Post date:** [14 April 2025 10:35 UTC](https://forum.confluent.io/t/mirrormaker2-not-replicating-certain-topics/37672/1 "2025-04-14T10:35:01Z")
Hello everyone! I have installed a MirrorMaker2 cluster with 2 nodes that is replicating a 3 broker + zookeeper Kafka cluster with just under 1100 topics. The replication works as expected for all but the following topics: STAGE.OU.task-updated-ics2-internal STAGE.OU.task-updated-internal STAGE.OU.task-upsert-events-internal STAGE.OU.task-upsert-ics2-events-internal TEST.OU.task-updated-ics2-internal TEST.OU.task-updated-internal TEST.OU.task-upsert-events-internal TEST.OU.task-upsert-ics2-events-internal There are several equivalent topics for a different environment that are replicated, however that is not important since they have no messages. I do not know if it is of any relevance but the message count is as follows: STAGE.OU.task-updated-ics2-internal - 55873 STAGE.OU.task-updated-internal - 420987 STAGE.OU.task-upsert-events-internal - 15000000+ STAGE.OU.task-upsert-ics2-events-internal - 43502 TEST.OU.task-updated-ics2-internal - 0 TEST.OU.task-updated-internal - 35 TEST.OU.task-upsert-events-internal - 14461 TEST.OU.task-upsert-ics2-events-internal - 0 I have made sure to include all topics through the mm2.properties configuration and that no topics are being excluded: ```auto topics=.* topics.exclude= ``` I have also tried replicating these topics alone without success: ```auto topics=STAGE.OU.task-updated-ics2-internal,STAGE.OU.task-updated-internal,STAGE.OU.task-upsert-events-internal,STAGE.OU.task-upsert-ics2-events-internal,TEST.OU.task-updated-ics2-internal,TEST.OU.task-updated-internal,TEST.OU.task-upsert-events-internal,TEST.OU.task-upsert-ics2-events-internal ``` I am using the IdentityReplicationPolicy replication policy class. There is no indication of an error in the MirrorMaker log. Source cluster is at version 7.6.5-ccs Target cluster is at version 7.6.4-ccs MirrorMaker cluster is at version 7.6.5-ccs Any idea why these topics are excluded from replication? ---
**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\ **Post date:** [14 April 2025 11:40 UTC](https://forum.confluent.io/t/mirrormaker2-not-replicating-certain-topics/37672/2 "2025-04-14T11:40:48Z")
hey @alexei.ivanov I’m not fully sure only thinking out loud: may this be related to the topic naming? as all the topic have _internal_ in their names… there is some logic around the internal topics in MM2 so maybe it’s somehow related to this ---
**Author:** ![alexei.ivanov](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/alexei.ivanov/32/5049_2.png) [@alexei.ivanov](https://forum.confluent.io/u/alexei.ivanov)\ **Post date:** [15 April 2025 10:35 UTC](https://forum.confluent.io/t/mirrormaker2-not-replicating-certain-topics/37672/3 "2025-04-15T10:35:55Z")
Hi @mmuehlbeyer! Thanks for a quick response. > may this be related to the topic naming? as all the topic have internal in their names… there is some logic around the internal topics in MM2 so maybe it’s somehow related to this That was my initial thought as well. However, I do see that topics that have the same name but different environment prefix (also end with _internal_ in their names) are being replicated just fine. So, I feel the theory is somewhat good, but not 100% accurate. Perhaps there is a glitch in this logic surrounding internal topics? Also, I have tried configuring `exclude.internal.topics` and setting it to false, but MirrorMaker seems to ignore this configuration no matter which way it is configured. Any other ideas/insights will be greatly appreciated! ---
**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\ **Post date:** [16 April 2025 11:22 UTC](https://forum.confluent.io/t/mirrormaker2-not-replicating-certain-topics/37672/4 "2025-04-16T11:22:50Z")
hmm pretty strange need to dig a bit around. maybe something worth to try what about a second MM2 config/cluster which only serves the topics not getting replicated at the moment? maybe we can gain some further insights around the behaviour with setting everything to debug then? (I’ve seen you already tried but probably within the same config/cluster?) ---
**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\ **Post date:** [16 April 2025 11:33 UTC](https://forum.confluent.io/t/mirrormaker2-not-replicating-certain-topics/37672/5 "2025-04-16T11:33:26Z")
another one : what happens if you switch to the default replication policy? I’ve just checked > according to that (line 97 following) you should not be hit by the internal topic check ```auto default boolean isMM2InternalTopic(String topic) { return topic.startsWith("mm2") && topic.endsWith(".internal") || isCheckpointsTopic(topic); } ``` ---
**Author:** ![alexei.ivanov](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/alexei.ivanov/32/5049_2.png) [@alexei.ivanov](https://forum.confluent.io/u/alexei.ivanov)\ **Post date:** [16 April 2025 15:40 UTC](https://forum.confluent.io/t/mirrormaker2-not-replicating-certain-topics/37672/6 "2025-04-16T15:40:54Z")
Thanks @mmuehlbeyer! After looking at the code a bit closer I may have discovered the issue. > according to that (line 97 following) you should not be hit by the internal topic check The code you referred to is a bit newer than the one I am currently running. The version was updated December 2024, so we need to update our clusters to version 7.8 or 7.9 (currently running 7.6.5). Take a look at this updated segment: [https://github.com/apache/kafka/commit/0bbed823e818a920fbaa2d54b50f4fcd81a8a759#diff-20a9931da3de4b577145af409d7e704e90ed30b410e4d8236e297b033dea6d0aR100-L109](https://github.com/apache/kafka/commit/0bbed823e818a920fbaa2d54b50f4fcd81a8a759#diff-20a9931da3de4b577145af409d7e704e90ed30b410e4d8236e297b033dea6d0aR100-L109) However, it still puzzles me how 2 of my 10 topics that end with “-internal” in their name got replicated. Anyway, I think I need to update the clusters before continuing finding out what is wrong. Thanks for pointing me in the right direction! Btw, DEBUG/TRACE log mode did not produce any leads. P.S. I am off on easter break so it might take a while before I can back to you with any results. ---
**Author:** ![alexei.ivanov](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/alexei.ivanov/32/5049_2.png) [@alexei.ivanov](https://forum.confluent.io/u/alexei.ivanov)\ **Post date:** [23 April 2025 12:55 UTC](https://forum.confluent.io/t/mirrormaker2-not-replicating-certain-topics/37672/7 "2025-04-23T12:55:20Z")
Hello again, Just updated the cluster to version 7.9.0 using but to no avail. It seem for some reason the code is not updated. I was unable to verify this by inspecting `DefaultReplicationPolicy` class, but the `DefaultTopicFilter` class still has the old topic [filter](https://github.com/apache/kafka/commit/0bbed823e818a920fbaa2d54b50f4fcd81a8a759#diff-6a5c6f348ad5fcab9e853cb2b4019e1fb3869d0993646b007cd5938ab834dfaaL36) as `topics.exclude` was still set to `[.*[\-\.]internal, .*\.replica, __.* ]` by default. This raises the obvious question: in what version of Confluent will [this feature](https://github.com/apache/kafka/pull/17815) be available? EDIT: KAFKA-17200 is part of Kafka version 4.0, so the question is actually about what version of Confluent will include Kafka 4.0? Thanks! ---
**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\ **Post date:** [27 April 2025 20:48 UTC](https://forum.confluent.io/t/mirrormaker2-not-replicating-certain-topics/37672/8 "2025-04-27T20:48:01Z")
Hi @alexei.ivanov I see thanks for providing the insights. as said one possible workaround might be to provide a custom replication policy. (or adapt the code and compile yourself, probably not cool if you need support or would like to use Confluent platform) Confluent plaform 8 will be built on Apache Kafka 4 > in preparation for the next Confluent Platform release built on Apache Kafka 4.0 > […] > Confluent Platform 8.0, coming in Summer 2025 for ref > **[New with Confluent Platform 7.9](https://www.confluent.io/blog/latest-confluent-platform/)** > > Confluent Platform 7.9 introduces the Oracle XStream CDC Connector, Client-Side Field Level Encryption (EA), Confluent for VS Code, and more. ---
**Author:** ![alexei.ivanov](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/alexei.ivanov/32/5049_2.png) [@alexei.ivanov](https://forum.confluent.io/u/alexei.ivanov)\ **Post date:** [2 May 2025 12:24 UTC](https://forum.confluent.io/t/mirrormaker2-not-replicating-certain-topics/37672/9 "2025-05-02T12:24:43Z")
Hi, @mmuehlbeyer! Thanks, although I was able to discover that page on my own I wanted to see if a more specific release date could be provided. We have considered the workarounds (code adaptation and custom policy) and came to the conclusion that they are not sustainable and are therefore too risky to implement. We ultimately decided to wait for the new version of Confluent to be released. But it is comforting to know that the workarounds we thought of were the same. Thanks again for your help! ---
**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/confluentcommunity/original/1X/c49438c90c9df282e9996fdf6971be890c71b65a.svg) [@system](https://forum.confluent.io/u/system)\ **Post date:** [9 May 2025 12:25 UTC](https://forum.confluent.io/t/mirrormaker2-not-replicating-certain-topics/37672/10 "2025-05-09T12:25:05Z")
Closed on 9 May 2025 12:25 UTC This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.