# Kafka streams not consuming after HPA trigger

**URL:** <https://forum.confluent.io/t/kafka-streams-not-consuming-after-hpa-trigger/8443>\
**Category:** Kafka Streams\
**Created:** [2 August 2023 10:36 UTC](https://forum.confluent.io/t/kafka-streams-not-consuming-after-hpa-trigger/8443 "2023-08-02T10:36:06Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![santosh.dhumale](https://avatars.discourse-cdn.com/v4/letter/s/cc9497/32.png) [@santosh.dhumale](https://forum.confluent.io/u/santosh.dhumale)\
**Post date:** [2 August 2023 10:36 UTC](https://forum.confluent.io/t/kafka-streams-not-consuming-after-hpa-trigger/8443/1 "2023-08-02T10:36:06Z")

</div>

Hi Team,

I am currently working on a project where we are using kafka stream and I am consuming few topics messages which get stored in GlobalKtable, KTable based on need. This application we are running in k8s and HPA strategy we mentioned as below.

When I deploy the application, both initial k8s pods consumes the topic message and partition divided equally 30 each (subscribed 60 partition topic).  
But when HPA kicks in and new pods get created, it does get partition assigned hence it remains ideal. I tried by changing max.poll.records and max.poll.interval.ms values but no luck.

Please note, the consumer group is same for all 3 k8s pod.

Did some one faced this issue?

hpa:  
minReplicaCount: 2 # this needs to be change to 4 for prod deployment  
maxReplicaCount: 3  
maxReplicaCountNonFullTraffic: 20  
averageValue: “700m”  
averageValueNonFullTraffic: “1400m”  
selectPolicyForScaleUp: Min  
scaleDown:  
stabilizationWindowSeconds: 1800  
policies:  
- type: Percent  
value: 10  
periodSeconds: 180  
- type: Pods  
value: 1  
periodSeconds: 180  
scaleUp:  
stabilizationWindowSeconds: 120  
policies:  
- type: Percent  
value: 30  
periodSeconds: 90  
- type: Pods  
value: 1  
periodSeconds: 90

---

<div class="post-metadata">

**Author:** ![mjsax](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mjsax/32/3113_2.png) [@mjsax](https://forum.confluent.io/u/mjsax)\
**Post date:** [2 August 2023 18:58 UTC](https://forum.confluent.io/t/kafka-streams-not-consuming-after-hpa-trigger/8443/2 "2023-08-02T18:58:32Z")

</div>

> [@santosh.dhumale](#):
>
> it does get partition assigned hence it remains ideal.

Do you mean, it does _not_ get partitioned assigned (and it’s just a typo)?

If that’s the case, it should have nothing to do with k8s but with Kafka Streams itself. The new instance should send a “join group request” to the broker, which would trigger a so-called rebalance to kick in. Such event should be logged broker and client side, so inspecting the logs seems to be a first good step.

Rebalancing is a very complex topic and you might want to check out this Kafka Summit talk about the latest state of the art: [Kafka Streams Rebalances and Assignments: The Whole Story with Alieh Saeedi & John Roesler](https://www.confluent.io/events/kafka-summit-london-2023/kafka-streams-rebalances-and-assignments-the-whole-story/)

---

<div class="post-metadata">

**Author:** ![santosh.dhumale](https://avatars.discourse-cdn.com/v4/letter/s/cc9497/32.png) [@santosh.dhumale](https://forum.confluent.io/u/santosh.dhumale)\
**Post date:** [3 August 2023 02:29 UTC](https://forum.confluent.io/t/kafka-streams-not-consuming-after-hpa-trigger/8443/3 "2023-08-03T02:29:24Z")

</div>

Thank @mjsax for looking into this. When I run my application with standalone jar everything works as expected but in K8s it doesnt.

I checked the debug logs for kakfa client and found it does send the joingroup request but coordinator does not assign any partition for same.

The strange thing is if I replace my stateStore with inMemory in that case every thing works

---

<div class="post-metadata">

**Author:** ![mjsax](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mjsax/32/3113_2.png) [@mjsax](https://forum.confluent.io/u/mjsax)\
**Post date:** [4 August 2023 00:53 UTC](https://forum.confluent.io/t/kafka-streams-not-consuming-after-hpa-trigger/8443/4 "2023-08-04T00:53:42Z")

</div>

Partition assignment is not done by the broker side group coordinator, but by one of the Kafka Streams instances, so you would need to check the Kafka Streams log (you need to find the right instance) of the so-called “group leader” that computes the assignment.

It’s weird that switching to in-memory changes the behavior; it should not have any impact…