Kafka consumer group rebalancing frequently

Clients
1

Hello ,
I have 20 pods running consumer . And consumer group is rebalancing frequently.
Kafka consumer is written in go lang
I have added the session.timeout as 45 sec and heartbeat interval is kept as default .

Any idea on why the rebalancing is happening frequently?

2

You need to figure out the reason for the rebalance… Does the group coordinator remove a member from the group due to missing heartbeat? Is poll() not called quickly enough and the consumer pro-actively send a leave-group request?

Broker and consumer logs should help you to find out.

3

hey,
i had some doubt on kafka consumer func StartConsumer() {
// Kafka consumer configuration
cfg := &kafka.ConfigMap{
“bootstrap.servers”: utils.GetConfig().KafkaConsumer.BootstrapServer,
“group.id”: utils.GetConfig().KafkaConsumer.KafkaGroupId,
“auto.offset.reset”: utils.GetConfig().KafkaConsumer.KafkaAutoOffsetReset,
“enable.auto.commit”: false,
“session.timeout.ms”: utils.GetConfig().KafkaConsumer.SessionTimeOut,
“heartbeat.interval.ms”: utils.GetConfig().KafkaConsumer.SessionTimeOut / 3,
“debug”: “all”,
}
this conifg ( “partition.assignment.strategy”: “sticky”, )
giving me config not found , how can i make htis consumer from range to sticky ?

the version of kafka i use : “github.com/confluentinc/confluent-kafka-go/v2/kafka”
(go lang)

4

The go client is base on librdkafak, which does not support “sticky” yet as it seems: librdkafka/CONFIGURATION.md at 876cf4a7e4a339fda8cea1c4a650c168a351a176 · confluentinc/librdkafka · GitHub

Even if I could find "cooperative-sticky" example: confluent-kafka-go/examples/cooperative_consumer_example/cooperative_consumer_example.go at cde2827bc49655eca0f9ce3fc1cda13cb6cdabc9 · confluentinc/confluent-kafka-go · GitHub

But I am not an expert on go-client.

1 Like