How can we make sure EoS in read-process scenario. read means we are reading from Kafka topic and doing some processing and then we are trying to commit the offset.
Lets suppose, we processed the messages but could not able to commit and before commit the process crashed. after restart, again trying to consume the same message. so how to handle such scenarios?
EoS functionality is controlled on the write side much like a database transaction: you write records in a transaction then commit the transaction at an appropriate safepoint. Reading data is slightly different to a database as Kafka is asynchronous meaning you use strongly ordered processing of events to ensure accurate state replication.
In your example, if you didn’t commit offsets when the process crashed, Kafka would provide you with the records again when you reconnected. If you did partial work that produdced side effects prior to the crash you’d need to ensure those side effects are idempotent. So for example, if your work writes to a database you could use a transaction to ensure that the side effect was rolled back if the process crashes or if you were say calling an external service you would need to use a unique identifier to ensure idempotence.
2 Likes
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.