Simple topic-to-topic copy in Mirror Maker 2

Hello there,

Is there a way to setup MM2 for a simple usage of just moving records between topics? We don’t want it to create the topics, update the configuration, etc. We already have a full setup to handle the definition and configuration of topics. We simply want to copy records from topic A into topic B. Sometimes A and B are on the same cluster, but most of the time they are on different clusters, and they often have the exact same name (no prefix or suffix).

I’ve tried using sync.topic.configs.enabled=false and use a KC transformer to define the target topic (with org.apache.kafka.connect.transforms.RegexRouter), but it still automatically creates another target topic (with the name of the source topic prefixed by the source alias).

The only way around it we’ve found is to define a custom replication policy, to tweak the names. However, we also had to define a custom SourceConnector extending the org.apache.kafka.connect.mirror.MirrorSourceConnector and override the isCycle method, or it fails when the topics have the same names on the two clusters.

That’s a lot of boilerplate for us, for a pretty basic use case. Are we missing something here? Is there an easier way to do this? Should we use something else than MM2? I couldn’t find another way to define a connector from Kafka to Kafka.

I forgot to mention that we were stuck in 2.8 for unrelated reasons, and I found out that there is some help for this starting in 3.0: as long as we keep the same topic name, we can use IdentityReplicationPolicy, and the MirrorSourceConnector handles having the same name for source and target topic.

There still isn’t a way to just define the target topic’s name, but it just needs a specific (and simple) replication-policy, nothing more. This replication-policy should be included in MM2, though - looks like a pretty typical use case.

1 Like

Hi Gael,

could you share the source code of your SourceConnector extending the org.apache.kafka.connect.mirror.MirrorSourceConnector? We have very similar requirement for topic migration purposes.

thank you


define the target topic’s name

You should be able to use RegexRouter for this

stuck on 2.8

Just download new version or use Docker… Newer MirrorMaker2 process will work with older brokers

couldn’t find another way to define a connector from Kafka to Kafka.

There are other connectors on Github, and there’s Confluent Replicator or Cluster Linking… You could also try Spark or Flink