Multiple consumers, single partition

Stream Processing
1

We have a Kafka cluster, and in it, we have a topic named ic_topic, which has 100 partitions.

Out of these partitions, we’re seeing that a partition is assigned to multiple consumers.

How is this possible ?

We’re using default Kafka CooperativeStickyAssignor partition assignment strategy.

props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, CooperativeStickyAssignor.class.getName());

Following is a redacted output of KafkaCLI showing two consumers attached to single partition.

GROUP         TOPIC      PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID                                                  HOST           CLIENT-ID
ic_group_v3   ic_topic   27         5814379         5814379         0    consumer-ic_group_v3-2-0cb7b077-xxxx-xxxx-xxxx-870e533a452e  /10.241.1.13   consumer-ic_group_v3-2
ic_group_v3   ic_topic   27         5814379         5814379         0    consumer-ic_group_v3-3-3101aee7-xxxx-xxxx-xxxx-e826b685cfdd  /10.5.1.12     consumer-ic_group_v3-3
2

If all consumers use the same group.id (as shown in the KafkaCLI output) this should not happen (assuming KafkaCLI shows the correct information).

Did you try to use bin/kafka-consumer-groups.sh to verify if KafkaCLI shows the correct information?

If the information is correct, on first glance, this might be a bug in the assignor. Can you reliable reproduce this issue? What version are you using? Did you try to use some different version?