First of all, I’m quite new to kafka.
I’m using Kafka 2.5.1 and Confluent 5.5.2
I have 2 instances of a stream app, with 4 stream threads each.
I am using localstate for performing aggregations.
So there are many topics that are read A,B,C. It’s values are transformed to an specific “dto” and sent to topic D. Topic D is consumed and a KTable is created for aggregating them. All have 24 partitions.
The problem is see is that while other topic partitions are evenly distributed among the different instances and stream threads. However topic D ( the one that is used to generate the aggregation ) partitions are all assigned to the same instance and just distributed among it’s stream threads.
So this is pretty unfortunate as this is one of the most resource intensive parts, so it would be much better and expected that topic D partitions be distributed among all the instances and streamthreads.
I have read that sticky assignor favors sending tasks to the same instances they were running before and this seems quite logical. That said i would expect it realizes that the assignment of tasks is totally unbalanced.
All other topics partitions are properly distributed/balanced.
Can someone point me in some reading or something that could allow me to address this?
thanks a lot