Offset commit failed on partition: The coordinator is not available

Jay · 5 February 2021 23:14

Hello,
We started seeing below error in the application logs(kafka-streams - v2.0.0) and since then not able to connect to the broker for consumption

ERROR [com.streams-18347f2b-69c8d4b59eb7-StreamThread-1] [Consumer clientId=com.streams-18347f2b-69c8d4b59eb7-StreamThread-1-consumer, groupId=com.streams] Offset commit failed on partition test-stream-topic-w-3 at offset 111224: The coordinator is not available. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)` When verified in the broker logs, following things have been observed : **1>** ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-29 (kafka.server.ReplicaManager)
ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-0 (kafka.server.ReplicaManager)
ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-18 (kafka.server.ReplicaManager)
ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-18 (kafka.server.ReplicaManager)
…
ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-44 (kafka.server.ReplicaManager)
ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-44 (kafka.server.ReplicaManager)
ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-19 (kafka.server.ReplicaManager)
ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-26 (kafka.server.ReplicaManager)
ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-26 (kafka.server.ReplicaManager) **2>** INFO [Partition test-stream-topic-w-3 broker=1] *Shrinking ISR from 1,9,10 to 1* . Leader: (highWatermark: 120324, endOffset: 120326). Out of sync replicas: (brokerId: 9, endOffset: 111222) (brokerId: 10, endOffset: 111224). (kafka.cluster.Partition) --

(observed lot of shrinking is happening on __consumer_offsets topic)

INFO [Partition test-stream-topic-w-3 broker=1] ISR updated to [1] and zkVersion updated to [535] (kafka.cluster.Partition)
**ERROR** [ReplicaManager broker=1] Error processing append operation on partition test-stream-topic-w-3 (kafka.server.ReplicaManager)
org.apache.kafka.common.errors. **NotEnoughReplicasException** : The size of the current ISR Set(1) is insufficient to satisfy the min.isr requirement of 2 for partition test-stream-topic-w-3

We’re using confluent v5.5.2
replication-factor on topic is 3

Server.properties :

    default.replication.factor=3
    min.insync.replicas=2
    offsets.topic.replication.factor=3
    replica.lag.time.max.ms=10000
    replica.fetch.wait.max.ms=5000

And we’re running a cluster of 7 brokers with 3 zk
Does anyone faced this kind of issue?
Thanks

mmuehlbeyer · 9 February 2021 14:27

Hi,

I assume all brokers are running?
What’s the output of
kafka-topics.sh --zookeeper <zookeeper_ip/hostname>:2181 --describe --topic __consumer_offsets

Jay · 10 February 2021 04:02

All the brokers are running at that point of time and only thing observed is - under-replication
Here’s the output of the describe command :

Topic: __consumer_offsets       PartitionCount: 50      ReplicationFactor: 3    Configs: compression.type=producer,cleanup.policy=compact,segment.bytes=104857600
        Topic: __consumer_offsets       Partition: 0    Leader: 2       Replicas: 2,1,11        Isr: 11,1,2
        Topic: __consumer_offsets       Partition: 1    Leader: 3       Replicas: 3,1,2 Isr: 3,1,2
        Topic: __consumer_offsets       Partition: 2    Leader: 5       Replicas: 5,1,2 Isr: 5,1,2
        Topic: __consumer_offsets       Partition: 3    Leader: 6       Replicas: 6,3,2 Isr: 6,3,2
        Topic: __consumer_offsets       Partition: 4    Leader: 3       Replicas: 3,1,2 Isr: 3,1,2
        Topic: __consumer_offsets       Partition: 5    Leader: 5       Replicas: 5,1,2 Isr: 5,1,2
        Topic: __consumer_offsets       Partition: 6    Leader: 3       Replicas: 3,1,2 Isr: 3,1,2
        Topic: __consumer_offsets       Partition: 7    Leader: 4       Replicas: 4,1,2 Isr: 4,1,2
        Topic: __consumer_offsets       Partition: 8    Leader: 5       Replicas: 5,1,2 Isr: 5,1,2
        Topic: __consumer_offsets       Partition: 9    Leader: 2       Replicas: 2,5,12        Isr: 5,12,2
        Topic: __consumer_offsets       Partition: 10   Leader: 1       Replicas: 1,4,2 Isr: 4,1,2
        Topic: __consumer_offsets       Partition: 11   Leader: 5       Replicas: 5,2,1 Isr: 5,1,2
        Topic: __consumer_offsets       Partition: 12   Leader: 3       Replicas: 3,10,7        Isr: 3,10,7
        Topic: __consumer_offsets       Partition: 13   Leader: 1       Replicas: 1,4,2 Isr: 4,1,2
        Topic: __consumer_offsets       Partition: 14   Leader: 1       Replicas: 1,4,2 Isr: 4,1,2
        Topic: __consumer_offsets       Partition: 15   Leader: 9       Replicas: 9,6,12        Isr: 9,6,12
        Topic: __consumer_offsets       Partition: 16   Leader: 2       Replicas: 2,1,3 Isr: 3,1,2
        Topic: __consumer_offsets       Partition: 17   Leader: 2       Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets       Partition: 18   Leader: 2       Replicas: 2,1,4 Isr: 4,1,2
        Topic: __consumer_offsets       Partition: 19   Leader: 2       Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets       Partition: 20   Leader: 3       Replicas: 3,1,2 Isr: 3,1,2
        Topic: __consumer_offsets       Partition: 21   Leader: 1       Replicas: 1,3,2 Isr: 3,1,2
        Topic: __consumer_offsets       Partition: 22   Leader: 1       Replicas: 1,2,3 Isr: 3,1,2
        Topic: __consumer_offsets       Partition: 23   Leader: 2       Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets       Partition: 24   Leader: 1       Replicas: 1,3,2 Isr: 3,1,2
        Topic: __consumer_offsets       Partition: 25   Leader: 2       Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets       Partition: 26   Leader: 2       Replicas: 2,1,4 Isr: 4,1,2
        Topic: __consumer_offsets       Partition: 27   Leader: 1       Replicas: 1,3,2 Isr: 3,1,2
        Topic: __consumer_offsets       Partition: 28   Leader: 3       Replicas: 3,1,6 Isr: 3,6,1
        Topic: __consumer_offsets       Partition: 29   Leader: 2       Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets       Partition: 30   Leader: 3       Replicas: 3,6,1 Isr: 3,6,1
        Topic: __consumer_offsets       Partition: 31   Leader: 2       Replicas: 2,5,12        Isr: 5,12,2
        Topic: __consumer_offsets       Partition: 32   Leader: 3       Replicas: 3,1,6 Isr: 3,6,1
        Topic: __consumer_offsets       Partition: 33   Leader: 3       Replicas: 3,1,6 Isr: 3,6,1
        Topic: __consumer_offsets       Partition: 34   Leader: 3       Replicas: 3,6,2 Isr: 3,6,2
        Topic: __consumer_offsets       Partition: 35   Leader: 3       Replicas: 3,2,1 Isr: 3,1,2
        Topic: __consumer_offsets       Partition: 36   Leader: 3       Replicas: 3,1,2 Isr: 3,1,2
        Topic: __consumer_offsets       Partition: 37   Leader: 2       Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets       Partition: 38   Leader: 2       Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets       Partition: 39   Leader: 11      Replicas: 11,2,3        Isr: 11,3,2
        Topic: __consumer_offsets       Partition: 40   Leader: 2       Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets       Partition: 41   Leader: 2       Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets       Partition: 42   Leader: 3       Replicas: 3,2,6 Isr: 3,6,2
        Topic: __consumer_offsets       Partition: 43   Leader: 2       Replicas: 2,1,4 Isr: 4,1,2
        Topic: __consumer_offsets       Partition: 44   Leader: 2       Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets       Partition: 45   Leader: 2       Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets       Partition: 46   Leader: 2       Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets       Partition: 47   Leader: 2       Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets       Partition: 48   Leader: 3       Replicas: 3,10,11       Isr: 3,10,11
        Topic: __consumer_offsets       Partition: 49   Leader: 5       Replicas: 5,2,1 Isr: 5,1,2

mmuehlbeyer · 10 February 2021 07:18

hmm ok I see what does a describe on the

test-stream-topic-w-3

show?
how are your broker ids configured?
I’m wondering about

Topic		Replies	Views
Error sending fetch request / Join group failed Kafka Streams	2	8624	18 May 2021
Error sending fetch request, Group coordinator is unavailable or invalid Ops	0	8313	4 October 2022
Manually committing offsets in Kafka using C# Non-Java Clients	1	6247	6 April 2022
Kafka Producer Error: Failed to Find Transaction Coordinator and Timed Out Waiting for Operation to Finish Clients	0	517	3 June 2024
Broker may not be available issue Clients	0	1572	27 October 2023

Offset commit failed on partition: The coordinator is not available

Related topics