Incoming messages limit

Clients
1

Hi,

We have a one producer who sends the millions of messages at one time. The issue we are facing is, kafka is able to handle this volumes and sending it to consumer at the same speed but our consumer application is not able to process same speed and messages are getting piled up at the consumer application side.

To address this issue, we would like to limit the messages speed for this particular producing topic or consumer topic. is this possible? if yes, what configuration changes we need to apply to limit the messages in producer/consumer topics?

looking forward to hearing the solution from community.

Regards,
Ravi

2

Which kind of consumer are you using? Typically a consumer polls for messages, processes those, and only then polls for the next. With max.poll.records the amount of messages for each poll can be limited.

3

Instead of artificially limiting the producer, I’d also check whether your consuming application is set up to scale its processing correctly and/or whether the Kafka topic has a sufficiently high number of partitions. For example, if the partition count is 30, you can run up to 30 parallel instances of your consuming application that collaboratively process the input data.