MirrorMaker 2 without internal topics

Ops Cluster Replication
1

We use MirrorMaker version 1 to mirror data back and forth between various Kafka clusters. The tool does not fulfill all wishes, but for us it is perfect (since we designed our system around MM1, since MM2 did not exist yet).

Since MM1 is considered obsolete and will be removed from the code base, we will have to switch to MM2. Basically this is not a problem, but a real difficulty are the new internal topics that are generated (heartbeats, mm2-configs, mm2-offsets, …).

In our system the MirrorMaker process (and also no other) is not allowed to create topics on any system. So is it really necessary for the MM2 to create these topics? Or is there also a possibility to run MM2 without these topics (maybe with reduced functionality, which I don’t need anyway)?

If it’s not possible, is it at least possible for the topics that don’t have cluster alias (e.g. heartbeats) to be used by multiple MM2 processes (with different configuration)?

By the way, from no case I will mirror any consumer group offsets. As far as I know MM2 does support this. But is there a way to disable this feature?

2

Already found out to disable some features I probably can live without:

refresh.topics.enabled=false
sync.group.offsets.enabled=false
refresh.groups.enabled=false
sync.topic.acls.enabled=false
emit.heartbeats.enabled=false
emit.checkpoints.enabled=false

Probably

sync.topic.configs.enabled=true

is all I need. Nevertheless a lot of internal topics are generated

‘source’ server

mm2-offset-syncs.target.internal

‘target’ server

__consumer_offsets
heartbeat
mm2-configs.source.internal
mm2-offsets.source.internal
mm2-status.source.internal
source.checkpoints.internal
3

The Kafka connect internal topics cannot be disabled.

Worth mentioning there are alternatives, too, such as LinkedIn Brooklyn, Uber uReplicator, Confluent Replicator, etc. Some of which may require additional services / topics, but may still suit your needs better than mm1/2.

I’ve also heard people using Flink for cluster replication as they can have more control over the code, although I assume they mean metric gathering or custom event handling more than byte-to-byte replication.

1 Like