Hi,
Is it a good idea to have multiple consumers having same group id and connecting to multiple topics?
My concern is that if one consumer is likely to fail more than others, the whole group will be in rebalance mode, and it will delay consuming for healthy consumers.
Hey @tarikh!
The issue you’re describing is relevant to consuming a single topic. I mean, when you have Topic1 and three consumers for it using Group1 group id, you would have the same problem, when you have one “problematic” consumer (i.e. Consumer1).
On the other hand, when you also have Topic2 and three other consumers, which use Group1 group id to consume from it, they should not be affected by group rebalancing caused by Consumer1 for Topic1.
However, I didn’t test it myself. Am I missing something?
This is the thing, I didn’t test my self either, was trying to find the answer in some Kafka documentation but didn’t find any.
Did you rely on specific Kafka document or guidelines ?
if so could you please share those?
this is totally normal if you have a consumer that has logic that depends on both topics. In fact Kafka has done a lot of work to make it so the rebalance isn’t as bad.
It really comes down to that first sentence, does your logic depend on two topics? If so you have to have a single consumer group on both topics.
KIP-415: Incremental Cooperative Rebalancing in Kafka Connect - Apache Kafka - Apache Software Foundation is the KIP you should for sure take a look at. Makes the rebalances much easier on SLAs.
As always, Gwen’s talk on the rebalance protocol is a must watch
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.