Imagine you have a simple topology, single input topic, one Key Value State Store utilizing RocksDB, single output topic.
Flow: Read from input topic, update KV Store with a count, write to output topic.
Imagine that this flow processes 40 messages very quickly and that these messages (40 to output) and (40 to changelog backing topic [assuming all unique keys] are being batched to be sent to the broker as a unit of work.
batch update: messages 1 - 40 → SUCCESS
This of course works as expected. If the application were to die, after this point, a new application instance would be able to continue because all data was in that successful batch commit.
Question: Is it ever possible that a batch of messages being committed would not contain all messages from a single topology execution? Recall, there are two commits required to successfully process the topology. One to state store changelog topic, one to output topic.
For example: Think about a particular message, let’s say it is message #35. Would it ever be possible to have the following batching occur?
batch update 1: messages 1 - 35 (changelog only) → SUCCESS
batch update 2: message 35 (output topic only) - message 40 → FAIL?
Why am I asking? There is a concern that State Store changelog updates might be committed independently of the output topic, if they are not always grouped in the same batch of commits to the broker. If they are split between batches, and the broker connection is lost, between those batches, then we would have an incorrect state when the “new” application instance is started.
We have looked and cannot see where batching of Topic and changeLog updates might occur in the code. I feel like we are missing something fundamental about Kafka. Is this handled somehow, behind the scenes?
Thanks for any help!