Issue in Distributing MirrorCheckpoint connector

Kafka Connect
1

Needed one help

NEW

9:24

Sai Venkatesh

I’m running a 5-node MirrorMaker2 in dedicated mode. In our case, the checkpoint connectors, which basically take care of syncing the consumer group offset, are not evenly distributed, but overall tasks, including heartbeat, source connectors, and checkpoint connectors, are properly scattered. Particularly, if you look at checkpoint, always one node is holding the majority of the connector, which is putting load on the CPU.

2

Let’s say I have 25 consumer groups in the source Kafka cluster and the Mirrormaker replicates the offsets to target. Out of the 5 connect nodes, only one node is handling all 25 CG offsets, and the rest 4 are all handling other source connectors.

3

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.