We recently created a Kafka cluster in prod with 6 broker nodes. Later we added 3 more brokers in the cluster. Now if are facing a problem while consuming the data. For some groupIds consumers are working fine. But for some groupIds consumers are not able to fetch any data from kafka. And we observed below logs on consumer.
and Additionally When we try to find the description of this group using the below command. We get timeout error.
Default replication factor is 3 for all the topics. Kafka process is running on all the nodes.
/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group contest-voltdb6fdsd-sdf-sdf-dsfs --describe
Error: Executing consumer group command failed due to org.apache.kafka.common.errors.TimeoutException: Call(callName=describeConsumerGroups, deadlineMs=1646983878839, tries=48, nextAllowedTryMs=1646983878940) timed out at 1646983878840 after 48 attempt(s)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=describeConsumerGroups, deadlineMs=1646983878839, tries=48, nextAllowedTryMs=1646983878940) timed out at 1646983878840 after 48 attempt(s)
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at kafka.admin.ConsumerGroupCommand$ConsumerGroupService.$anonfun$describeConsumerGroups$1(ConsumerGroupCommand.scala:562)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:273)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.collection.TraversableLike.map(TraversableLike.scala:273)
at scala.collection.TraversableLike.map$(TraversableLike.scala:266)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at kafka.admin.ConsumerGroupCommand$ConsumerGroupService.describeConsumerGroups(ConsumerGroupCommand.scala:561)
at kafka.admin.ConsumerGroupCommand$ConsumerGroupService.collectGroupsOffsets(ConsumerGroupCommand.scala:577)
at
Logs on consumer:
13:00:24.364 [main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] (Re-)joining group
13:00:24.364 [main] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Sending JoinGroup ((type: JoinGroupRequest, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs, sessionTimeout=10000, rebalanceTimeout=20000, memberId=, protocolType=consumer, groupProtocols=org.apache.kafka.common.requests.JoinGroupRequest$ProtocolMetadata@757529a4)) to coordinator 172.23.195.6:9092 (id: 2147482645 rack: null)
13:00:24.579 [main] DEBUG org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Created socket with SO_RCVBUF = 66787, SO_SNDBUF = 132211, SO_TIMEOUT = 0 to node 2147482645
13:00:24.580 [main] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Completed connection to node 2147482645. Fetching API versions.
13:00:24.580 [main] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Initiating API versions fetch from node 2147482645.
13:00:24.797 [main] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Recorded API versions for node 2147482645: (Produce(0): 0 to 8 [usable: 7], Fetch(1): 0 to 11 [usable: 10], ListOffsets(2): 0 to 5 [usable: 5], Metadata(3): 0 to 9 [usable: 7], LeaderAndIsr(4): 0 to 4 [usable: 2], StopReplica(5): 0 to 3 [usable: 1], UpdateMetadata(6): 0 to 6 [usable: 5], ControlledShutdown(7): 0 to 3 [usable: 2], OffsetCommit(8): 0 to 8 [usable: 6], OffsetFetch(9): 0 to 7 [usable: 5], FindCoordinator(10): 0 to 3 [usable: 2], JoinGroup(11): 0 to 7 [usable: 4], Heartbeat(12): 0 to 4 [usable: 2], LeaveGroup(13): 0 to 4 [usable: 2], SyncGroup(14): 0 to 5 [usable: 2], DescribeGroups(15): 0 to 5 [usable: 2], ListGroups(16): 0 to 4 [usable: 2], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 3 [usable: 2], CreateTopics(19): 0 to 5 [usable: 3], DeleteTopics(20): 0 to 4 [usable: 3], DeleteRecords(21): 0 to 2 [usable: 1], InitProducerId(22): 0 to 3 [usable: 1], OffsetForLeaderEpoch(23): 0 to 3 [usable: 2], AddPartitionsToTxn(24): 0 to 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1 [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to 3 [usable: 2], DescribeAcls(29): 0 to 2 [usable: 1], CreateAcls(30): 0 to 2 [usable: 1], DeleteAcls(31): 0 to 2 [usable: 1], DescribeConfigs(32): 0 to 3 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1], AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 2 [usable: 1], SaslAuthenticate(36): 0 to 2 [usable: 1], CreatePartitions(37): 0 to 2 [usable: 1], CreateDelegationToken(38): 0 to 2 [usable: 1], RenewDelegationToken(39): 0 to 2 [usable: 1], ExpireDelegationToken(40): 0 to 2 [usable: 1], DescribeDelegationToken(41): 0 to 2 [usable: 1], DeleteGroups(42): 0 to 2 [usable: 1], ElectPreferredLeaders(43): 0 to 2 [usable: 0], UNKNOWN(44): 0 to 1, UNKNOWN(45): 0, UNKNOWN(46): 0, UNKNOWN(47): 0, UNKNOWN(48): 0, UNKNOWN(49): 0)
13:00:25.008 [main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Group coordinator 172.23.195.6:9092 (id: 2147482645 rack: null) is unavailable or invalid, will attempt rediscovery
13:00:25.008 [main] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Attempt to join group failed due to obsolete coordinator information: This is not the correct coordinator.
13:00:25.020 [main] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Sending FindCoordinator request to broker 172.23.210.227:9092 (id: 1004 rack: us-east-1c)
13:00:25.021 [main] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Manually disconnected from 2147482645. Removed requests: .
13:00:25.287 [main] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Received FindCoordinator response ClientResponse(receivedTimeMs=1646983825287, latencyMs=266, disconnected=false, requestHeader=RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=2, clientId=consumer-1, correlationId=1014), responseBody=FindCoordinatorResponse(throttleTimeMs=0, errorMessage='NONE', error=NONE, node=172.23.195.6:9092 (id: 1002 rack: null)))
13:00:25.287 [main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Discovered group coordinator 172.23.195.6:9092 (id: 2147482645 rack: null)
13:00:25.287 [main] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Initiating connection to node 172.23.195.6:9092 (id: 2147482645 rack: null) using address /172.23.195.6
13:00:25.288 [main] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Disabling heartbeat thread
13:00:25.288 [main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] (Re-)joining group
13:00:25.288 [main] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Sending JoinGroup ((type: JoinGroupRequest, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs, sessionTimeout=10000, rebalanceTimeout=20000, memberId=, protocolType=consumer, groupProtocols=org.apache.kafka.common.requests.JoinGroupRequest$ProtocolMetadata@5c41d037)) to coordinator 172.23.195.6:9092 (id: 2147482645 rack: null)
13:00:25.580 [main] DEBUG org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Created socket with SO_RCVBUF = 66787, SO_SNDBUF = 132211, SO_TIMEOUT = 0 to node 2147482645
13:00:25.580 [main] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Completed connection to node 2147482645. Fetching API versions.
13:00:25.580 [main] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Initiating API versions fetch from node 2147482645.
13:00:25.840 [main] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Recorded API versions for node 2147482645: (Produce(0): 0 to 8 [usable: 7], Fetch(1): 0 to 11 [usable: 10], ListOffsets(2): 0 to 5 [usable: 5], Metadata(3): 0 to 9 [usable: 7], LeaderAndIsr(4): 0 to 4 [usable: 2], StopReplica(5): 0 to 3 [usable: 1], UpdateMetadata(6): 0 to 6 [usable: 5], ControlledShutdown(7): 0 to 3 [usable: 2], OffsetCommit(8): 0 to 8 [usable: 6], OffsetFetch(9): 0 to 7 [usable: 5], FindCoordinator(10): 0 to 3 [usable: 2], JoinGroup(11): 0 to 7 [usable: 4], Heartbeat(12): 0 to 4 [usable: 2], LeaveGroup(13): 0 to 4 [usable: 2], SyncGroup(14): 0 to 5 [usable: 2], DescribeGroups(15): 0 to 5 [usable: 2], ListGroups(16): 0 to 4 [usable: 2], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 3 [usable: 2], CreateTopics(19): 0 to 5 [usable: 3], DeleteTopics(20): 0 to 4 [usable: 3], DeleteRecords(21): 0 to 2 [usable: 1], InitProducerId(22): 0 to 3 [usable: 1], OffsetForLeaderEpoch(23): 0 to 3 [usable: 2], AddPartitionsToTxn(24): 0 to 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1 [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to 3 [usable: 2], DescribeAcls(29): 0 to 2 [usable: 1], CreateAcls(30): 0 to 2 [usable: 1], DeleteAcls(31): 0 to 2 [usable: 1], DescribeConfigs(32): 0 to 3 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1], AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 2 [usable: 1], SaslAuthenticate(36): 0 to 2 [usable: 1], CreatePartitions(37): 0 to 2 [usable: 1], CreateDelegationToken(38): 0 to 2 [usable: 1], RenewDelegationToken(39): 0 to 2 [usable: 1], ExpireDelegationToken(40): 0 to 2 [usable: 1], DescribeDelegationToken(41): 0 to 2 [usable: 1], DeleteGroups(42): 0 to 2 [usable: 1], ElectPreferredLeaders(43): 0 to 2 [usable: 0], UNKNOWN(44): 0 to 1, UNKNOWN(45): 0, UNKNOWN(46): 0, UNKNOWN(47): 0, UNKNOWN(48): 0, UNKNOWN(49): 0)
13:00:26.053 [main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Group coordinator 172.23.195.6:9092 (id: 2147482645 rack: null) is unavailable or invalid, will attempt rediscovery
13:00:26.053 [main] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Attempt to join group failed due to obsolete coordinator information: This is not the correct coordinator.
13:00:26.053 [main] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Sending FindCoordinator request to broker 172.23.210.227:9092 (id: 1004 rack: us-east-1c)
13:00:26.053 [main] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Manually disconnected from 2147482645. Removed requests: .
13:00:26.265 [main] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Received FindCoordinator response ClientResponse(receivedTimeMs=1646983826264, latencyMs=211, disconnected=false, requestHeader=RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=2, clientId=consumer-1, correlationId=1017), responseBody=FindCoordinatorResponse(throttleTimeMs=0, errorMessage='NONE', error=NONE, node=172.23.195.6:9092 (id: 1002 rack: null)))
13:00:26.265 [main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Discovered group coordinator 172.23.195.6:9092 (id: 2147482645 rack: null)
13:00:26.265 [main] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Initiating connection to node 172.23.195.6:9092 (id: 2147482645 rack: null) using address /172.23.195.6
13:00:26.265 [main] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Disabling heartbeat thread
13:00:26.265 [main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] (Re-)joining group
13:00:26.265 [main] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Sending JoinGroup ((type: JoinGroupRequest, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs, sessionTimeout=10000, rebalanceTimeout=20000, memberId=, protocolType=consumer, groupProtocols=org.apache.kafka.common.requests.JoinGroupRequest$ProtocolMetadata@5ec77191)) to coordinator 172.23.195.6:9092 (id: 2147482645 rack: null)
13:00:26.509 [main] DEBUG org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Created socket with SO_RCVBUF = 66787, SO_SNDBUF = 132211, SO_TIMEOUT = 0 to node 2147482645
13:00:26.509 [main] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Completed connection to node 2147482645. Fetching API versions.
13:00:26.509 [main] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Initiating API versions fetch from node 2147482645.
13:00:26.818 [main] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Recorded API versions for node 2147482645: (Produce(0): 0 to 8 [usable: 7], Fetch(1): 0 to 11 [usable: 10], ListOffsets(2): 0 to 5 [usable: 5], Metadata(3): 0 to 9 [usable: 7], LeaderAndIsr(4): 0 to 4 [usable: 2], StopReplica(5): 0 to 3 [usable: 1], UpdateMetadata(6): 0 to 6 [usable: 5], ControlledShutdown(7): 0 to 3 [usable: 2], OffsetCommit(8): 0 to 8 [usable: 6], OffsetFetch(9): 0 to 7 [usable: 5], FindCoordinator(10): 0 to 3 [usable: 2], JoinGroup(11): 0 to 7 [usable: 4], Heartbeat(12): 0 to 4 [usable: 2], LeaveGroup(13): 0 to 4 [usable: 2], SyncGroup(14): 0 to 5 [usable: 2], DescribeGroups(15): 0 to 5 [usable: 2], ListGroups(16): 0 to 4 [usable: 2], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 3 [usable: 2], CreateTopics(19): 0 to 5 [usable: 3], DeleteTopics(20): 0 to 4 [usable: 3], DeleteRecords(21): 0 to 2 [usable: 1], InitProducerId(22): 0 to 3 [usable: 1], OffsetForLeaderEpoch(23): 0 to 3 [usable: 2], AddPartitionsToTxn(24): 0 to 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1 [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to 3 [usable: 2], DescribeAcls(29): 0 to 2 [usable: 1], CreateAcls(30): 0 to 2 [usable: 1], DeleteAcls(31): 0 to 2 [usable: 1], DescribeConfigs(32): 0 to 3 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1], AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 2 [usable: 1], SaslAuthenticate(36): 0 to 2 [usable: 1], CreatePartitions(37): 0 to 2 [usable: 1], CreateDelegationToken(38): 0 to 2 [usable: 1], RenewDelegationToken(39): 0 to 2 [usable: 1], ExpireDelegationToken(40): 0 to 2 [usable: 1], DescribeDelegationToken(41): 0 to 2 [usable: 1], DeleteGroups(42): 0 to 2 [usable: 1], ElectPreferredLeaders(43): 0 to 2 [usable: 0], UNKNOWN(44): 0 to 1, UNKNOWN(45): 0, UNKNOWN(46): 0, UNKNOWN(47): 0, UNKNOWN(48): 0, UNKNOWN(49): 0)
13:00:27.129 [main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=contest-voltdb6fdsd-sdf-sdf-dsfs] Group coordinator 172.23.195.6:9092 (id: 21