I have 3 kafka-connect nodes, running mm2 tasks.
Those task replicate 4 topics from remote uk and us clusters into single uk kafka cluster.
Everything works fine most of the time, but at some point I noticed that the lag on 1 of the topics started increasing.
Watching the logs, I’ve noticed this in the beginning of the lag growing:
[2023-10-21 05:05:56,820] INFO [mm2-msc-us-dc1-2-live|task-19] [Consumer clientId=mm2-us-dc1-2-live->mm2-kafka|mm2-msc-us-dc1-2-live-19|replication-consumer, groupId=null] Disconnecting from node 895 due to request timeout. (org.apache.kafka.clients.NetworkClient:836)
[2023-10-21 05:05:56,821] INFO [mm2-msc-us-dc1-2-live|task-19] [Consumer clientId=mm2-us-dc1-2-live->mm2-kafka|mm2-msc-us-dc1-2-live-19|replication-consumer, groupId=null] Cancelled in-flight FETCH request with correlation id 1215447 due to node 895 being disconnected (elapsed time since creation: 30025ms, elapsed time since send: 30025ms, request timeout: 30000ms) (org.apache.kafka.clients.NetworkClient:344)
[2023-10-21 05:05:56,821] INFO [mm2-msc-us-dc1-2-live|task-19] [Consumer clientId=mm2-us-dc1-2-live->mm2-kafka|mm2-msc-us-dc1-2-live-19|replication-consumer, groupId=null] Error sending fetch request (sessionId=1799400446, epoch=1214974) to node 895: (org.apache.kafka.clients.FetchSessionHandler:615)
org.apache.kafka.common.errors.DisconnectException
the chart with lag (it uses +3 offset, so literally it started at the same time as in logs above)
Any idea what can be the reason? Why it stopped replicating only 1 topic and why it fixed itself in 1 hour?