I’m exploring the Confluent Parallel Consumer library (parallel-consumer) for message processing in Kafka.
This library offers three levels of parallelization: key-level, unordered, and partition-level.
My question is:
Offset Persistence: Do all three parallelism types persist the committed offsets to the Kafka broker? This ensures that after a failure or rebalancing, only unprocessed messages are replayed, regardless of the consumer’s current offset.
Offset Commit Frequency: How often are offsets committed to the broker? Frequent commits might lead to duplicate processing. To avoid this, it’s recommended to design idempotent consumers.
Retries and Delayed Retries: The documentation mentions retries and delayed retries (retry document). I’d like to understand how these work:
- Are messages marked for retry stored in the broker or retried in-memory?
- How are delayed retries handled internally? How are these messages persisted and managed for delayed retries?
Any insights or pointers would be greatly appreciated!