Hi I have a Kafka broker with one topic. I need to more consumers read from the same topic. I can’t find any tutorial, document helpful for me.
Any help is welcome.
Thanks.
PS: I’m using python.
There isn’t a tutorial on this. All that you’ll need to do is configure your apps(clients) to have different consumer groups. They will each read the same messages from the same topic(s), they can read the messages at different rates.
The Magical Group Coordination Protocol of Apache Kafka - YouTube @gwenshap does a great job explaining this concept there.
2 Likes
Well that depends of course on if he need more consumers to handle the load, AR needs to process the same messages with different logic.
For scaling you need the same consumer group, and make sure you have at least as many partitions as you want/need to run consumer instances. For processing the same with different code, you need to have different consumer groups.
Thank you for your suggestions: it seems my response is to have n partitions of a topic as many as the consumer, so each consumer read from a partition of the fixed topic.
Cheers.
Giuseppe
2 Likes
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.