# Offset commit failed on partition: The coordinator is not available

**URL:** <https://forum.confluent.io/t/offset-commit-failed-on-partition-the-coordinator-is-not-available/723>\
**Category:** Kafka Streams\
**Created:** [5 February 2021 23:14 UTC](https://forum.confluent.io/t/offset-commit-failed-on-partition-the-coordinator-is-not-available/723 "2021-02-05T23:14:46Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jay](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/jay/32/804_2.png) [@Jay](https://forum.confluent.io/u/Jay)\
**Post date:** [5 February 2021 23:14 UTC](https://forum.confluent.io/t/offset-commit-failed-on-partition-the-coordinator-is-not-available/723/1 "2021-02-05T23:14:46Z")

</div>

Hello,  
We started seeing below error in the application logs( **kafka-streams - v2.0.0** ) and since then not able to connect to the broker for consumption

```auto
ERROR [com.streams-18347f2b-69c8d4b59eb7-StreamThread-1] [Consumer clientId=com.streams-18347f2b-69c8d4b59eb7-StreamThread-1-consumer, groupId=com.streams] Offset commit failed on partition test-stream-topic-w-3 at offset 111224: The coordinator is not available. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)` When verified in the broker logs, following things have been observed : **1>** ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-29 (kafka.server.ReplicaManager)
ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-0 (kafka.server.ReplicaManager)
ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-18 (kafka.server.ReplicaManager)
ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-18 (kafka.server.ReplicaManager)
…
ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-44 (kafka.server.ReplicaManager)
ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-44 (kafka.server.ReplicaManager)
ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-19 (kafka.server.ReplicaManager)
ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-26 (kafka.server.ReplicaManager)
ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-26 (kafka.server.ReplicaManager) **2>** INFO [Partition test-stream-topic-w-3 broker=1] *Shrinking ISR from 1,9,10 to 1* . Leader: (highWatermark: 120324, endOffset: 120326). Out of sync replicas: (brokerId: 9, endOffset: 111222) (brokerId: 10, endOffset: 111224). (kafka.cluster.Partition) --

```

**(observed lot of shrinking is happening on \_\_consumer\_offsets topic)**

```auto
INFO [Partition test-stream-topic-w-3 broker=1] ISR updated to [1] and zkVersion updated to [535] (kafka.cluster.Partition)
**ERROR** [ReplicaManager broker=1] Error processing append operation on partition test-stream-topic-w-3 (kafka.server.ReplicaManager)
org.apache.kafka.common.errors. **NotEnoughReplicasException** : The size of the current ISR Set(1) is insufficient to satisfy the min.isr requirement of 2 for partition test-stream-topic-w-3

```

We’re using **confluent v5.5.2**  
`replication-factor on topic is 3`

**Server.properties :**

```auto
    default.replication.factor=3
    min.insync.replicas=2
    offsets.topic.replication.factor=3
    replica.lag.time.max.ms=10000
    replica.fetch.wait.max.ms=5000

```

And we’re running a cluster of `7 brokers with 3 zk`  
Does anyone faced this kind of issue?  
Thanks

---

<div class="post-metadata">

**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\
**Post date:** [9 February 2021 14:27 UTC](https://forum.confluent.io/t/offset-commit-failed-on-partition-the-coordinator-is-not-available/723/2 "2021-02-09T14:27:04Z")

</div>

Hi,

I assume all brokers are running?  
What’s the output of  
`kafka-topics.sh --zookeeper <zookeeper_ip/hostname>:2181 --describe --topic __consumer_offsets`

---

<div class="post-metadata">

**Author:** ![Jay](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/jay/32/804_2.png) [@Jay](https://forum.confluent.io/u/Jay)\
**Post date:** [10 February 2021 04:02 UTC](https://forum.confluent.io/t/offset-commit-failed-on-partition-the-coordinator-is-not-available/723/3 "2021-02-10T04:02:39Z")

</div>

All the brokers are running at that point of time and only thing observed is - `under-replication`  
Here’s the output of the describe command :

```
Topic: __consumer_offsets PartitionCount: 50 ReplicationFactor: 3 Configs: compression.type=producer,cleanup.policy=compact,segment.bytes=104857600
        Topic: __consumer_offsets Partition: 0 Leader: 2 Replicas: 2,1,11 Isr: 11,1,2
        Topic: __consumer_offsets Partition: 1 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
        Topic: __consumer_offsets Partition: 2 Leader: 5 Replicas: 5,1,2 Isr: 5,1,2
        Topic: __consumer_offsets Partition: 3 Leader: 6 Replicas: 6,3,2 Isr: 6,3,2
        Topic: __consumer_offsets Partition: 4 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
        Topic: __consumer_offsets Partition: 5 Leader: 5 Replicas: 5,1,2 Isr: 5,1,2
        Topic: __consumer_offsets Partition: 6 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
        Topic: __consumer_offsets Partition: 7 Leader: 4 Replicas: 4,1,2 Isr: 4,1,2
        Topic: __consumer_offsets Partition: 8 Leader: 5 Replicas: 5,1,2 Isr: 5,1,2
        Topic: __consumer_offsets Partition: 9 Leader: 2 Replicas: 2,5,12 Isr: 5,12,2
        Topic: __consumer_offsets Partition: 10 Leader: 1 Replicas: 1,4,2 Isr: 4,1,2
        Topic: __consumer_offsets Partition: 11 Leader: 5 Replicas: 5,2,1 Isr: 5,1,2
        Topic: __consumer_offsets Partition: 12 Leader: 3 Replicas: 3,10,7 Isr: 3,10,7
        Topic: __consumer_offsets Partition: 13 Leader: 1 Replicas: 1,4,2 Isr: 4,1,2
        Topic: __consumer_offsets Partition: 14 Leader: 1 Replicas: 1,4,2 Isr: 4,1,2
        Topic: __consumer_offsets Partition: 15 Leader: 9 Replicas: 9,6,12 Isr: 9,6,12
        Topic: __consumer_offsets Partition: 16 Leader: 2 Replicas: 2,1,3 Isr: 3,1,2
        Topic: __consumer_offsets Partition: 17 Leader: 2 Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets Partition: 18 Leader: 2 Replicas: 2,1,4 Isr: 4,1,2
        Topic: __consumer_offsets Partition: 19 Leader: 2 Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets Partition: 20 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
        Topic: __consumer_offsets Partition: 21 Leader: 1 Replicas: 1,3,2 Isr: 3,1,2
        Topic: __consumer_offsets Partition: 22 Leader: 1 Replicas: 1,2,3 Isr: 3,1,2
        Topic: __consumer_offsets Partition: 23 Leader: 2 Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets Partition: 24 Leader: 1 Replicas: 1,3,2 Isr: 3,1,2
        Topic: __consumer_offsets Partition: 25 Leader: 2 Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets Partition: 26 Leader: 2 Replicas: 2,1,4 Isr: 4,1,2
        Topic: __consumer_offsets Partition: 27 Leader: 1 Replicas: 1,3,2 Isr: 3,1,2
        Topic: __consumer_offsets Partition: 28 Leader: 3 Replicas: 3,1,6 Isr: 3,6,1
        Topic: __consumer_offsets Partition: 29 Leader: 2 Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets Partition: 30 Leader: 3 Replicas: 3,6,1 Isr: 3,6,1
        Topic: __consumer_offsets Partition: 31 Leader: 2 Replicas: 2,5,12 Isr: 5,12,2
        Topic: __consumer_offsets Partition: 32 Leader: 3 Replicas: 3,1,6 Isr: 3,6,1
        Topic: __consumer_offsets Partition: 33 Leader: 3 Replicas: 3,1,6 Isr: 3,6,1
        Topic: __consumer_offsets Partition: 34 Leader: 3 Replicas: 3,6,2 Isr: 3,6,2
        Topic: __consumer_offsets Partition: 35 Leader: 3 Replicas: 3,2,1 Isr: 3,1,2
        Topic: __consumer_offsets Partition: 36 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
        Topic: __consumer_offsets Partition: 37 Leader: 2 Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets Partition: 38 Leader: 2 Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets Partition: 39 Leader: 11 Replicas: 11,2,3 Isr: 11,3,2
        Topic: __consumer_offsets Partition: 40 Leader: 2 Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets Partition: 41 Leader: 2 Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets Partition: 42 Leader: 3 Replicas: 3,2,6 Isr: 3,6,2
        Topic: __consumer_offsets Partition: 43 Leader: 2 Replicas: 2,1,4 Isr: 4,1,2
        Topic: __consumer_offsets Partition: 44 Leader: 2 Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets Partition: 45 Leader: 2 Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets Partition: 46 Leader: 2 Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets Partition: 47 Leader: 2 Replicas: 2,4,1 Isr: 4,1,2
        Topic: __consumer_offsets Partition: 48 Leader: 3 Replicas: 3,10,11 Isr: 3,10,11
        Topic: __consumer_offsets Partition: 49 Leader: 5 Replicas: 5,2,1 Isr: 5,1,2

```

---

<div class="post-metadata">

**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\
**Post date:** [10 February 2021 07:18 UTC](https://forum.confluent.io/t/offset-commit-failed-on-partition-the-coordinator-is-not-available/723/4 "2021-02-10T07:18:11Z")

</div>

hmm ok I see what does a describe on the

```auto
test-stream-topic-w-3

```

show?  
how are your broker ids configured?  
I’m wondering about

> [@Jay](#):
>
> ` Topic: __consumer_offsets Partition: 15 Leader: 9 Replicas: 9,6,12 Isr: 9,6,12`

> [@Jay](#):
>
> ` Topic: __consumer_offsets Partition: 39 Leader: 11 Replicas: 11,2,3 Isr: 11,3,2`