We are using kafka streams in kubernetes deployment and running 10 instance of our pod. All the pod has static consumer group ID configured. Each pod has 4 streaming thread. The topic has 60 partition.
We are seeing rebalance storm and message not getting processed / lag being built. Again this problem is very much reproducible.
From kafka properties point of view
max.poll.inteval.ms is set to Int.MaxValue
session.timeout.ms is set to 60 sec
heartbeat.interval.ms is 20 sec.
Rebalance itself may be fine but we are seeing messages are not processed for long time and huge lag getting built. Please help how to overcome this issue.