Kafka offers the configuration properties of:
fetch.max.wait.ms (and fetch.min.bytes that relates to it)
This configurations is enforced by the broker, and therefor only useful when the topic has 1 partition. In the case of multiple partitions, one broker can no longer enforce this configuration and it doesn’t work.
I want to ask if there is an implementation on the consumer side, meaning:
- If a batch is bigger than size X - continue with the batch
- if a batch is smaller than size X but Y time has passed since a batch was read - continue with the batch
- if the batch is smaller than size X and time is less than Y, mark the batch as unread (so messages will be taken again) and sleep for some time before trying again.