Hi,
I’ve Kafka consumer with maximum fetch size per partition (max.partition.fetch.bytes ) set to 5MB. Fetches respect this until the first pause and resume consuming from partition. This cause the fetch request size to increase up to 30MB. I have no clue what is causing this behavior. Does anyone have any idea?
Here is a plot of fetch-size-avg metric for visualization of described behavior.
No idea. What version of the clients/broker are you using?
Thanks for the reply. I’m using broker and client version 3.8. Maybe, it can be due to a misunderstanding in reporting, because if client has any completed fetch requests to load a partition that has been pause, responses are not discarded, but remain in fetch buffer for future poll execution. Let’s say, client has had 2 inflight requests to fetch partition 0. Then this partition was paused, so it leads in client having two completed fetch requests in buffer, which cannot be skipped until resume is called. And the metric is updated after the fetch request is consumed by poll.
Anyway, I suppose my theory is wrong, but I have no other reasonable explanation for this behavior. I’ll try to reproduce it and look at the network traffic and broker metrics. If you have any other idea, I would be glad to hear it.
