Hi all,
We have an unusual situation in which a consumer group for a topic does not seem to be consuming any messages that are written to that topic. I can see from prometheus that LogEndOffset - LogStartOffset = 4 for this topic so there should be 4 messages to consume.
We have enabled debug logging for the affected service and are able to see that the service appears to be polling a kafka broker and requesting new messages:
Received FETCH response from node 4...
Node 0 sent an incremental fetch response with throttleTimeMs = 4 for session 963447090 with 0 response partition(s), 1 implied partition(s)...
Added READ_UNCOMMITTED fetch request for partition xxx at position...
Built incremental fetch...
Sending READ_UNCOMMITTED IncrementalFetchRequest...
Sending FETCH request with header...
We don’t see any errors, and the logging is similar to the logs produced when consuming from other unaffected topics.
We previously had exactly_once_beta delivery semantics configured however we switched to at_least_once as we noticed some issues consuming from other topics with exactly_once_beta.
We have tried changing the consumer group id (after changing from exactly_once_beta → at_least_once), however this has not made a difference to the affected topic.
Any advice / suggestions for other things to try?
We are using:
Kafka 2.6.0
micronaut-kafka 3.1.0
micronaut-kafka-streams 3.1.0.