Changelog restoration with persistent storage

Kafka Streams
1

In the past, restoring state stores from the changelog topics at startup took a lot of time. That’s why we switched from stateless deployments to stateful sets with persistent volume claims. That, of course, speeds up the startup significantly.

In some situations we clean a changelog in order to repopulate it:

  1. some source topic contains incorrect events and the publishing system repopulates the topic instead of sending tombstone events
  2. the type of the changelog’s payloads has changed

So, what happens when the state stores are restored from persistent storage? Will the Kafka Streams application notice that the persisted events are stale and clear the state store in accordance with the cleared changelog?

We already faced the issue that after a initial load of some source topic (case 1 above) we had invalid events in our output topic even though the invalid event was no longer in the source topic. That’s why we suspected some intermediate value might have restored from persisted state stores.

2

So, what happens when the state stores are restored from persistent storage? Will the Kafka Streams application notice that the persisted events are stale and clear the state store in accordance with the cleared changelog?

No. When you clean the changelog, you should also delete the corresponding RocksDB state on local disk.

3

Thanks Matthias

That confirms our suspicion. So, in order to do a proper initial load that includes the repopulation of intermediate changelog topics we need to

  • switch the application in a non-consuming mode
  • clear all affected changelog topics
  • delete corresponding rocksdb directories on the different pods
  • reset offset for initial load
  • switch the application back in consuming mode

This seems rather high-maintenance, are there better approaches?

I looked at Kafka Streams Application Reset Tool and data reprocessing with kafka streams: resetting a streams application, but the application reset tool

  1. seems to delete all internal topics, while often a majority of them should remain
  2. still requires us to delete the rocksdb directories where we use persistent storage for state stores.

To me, the application reset tools seems to be a convenient way to reset the complete application/topology. But if we need to reset only a single path from an input topic through the topology, the manual way is the way to go. Or did I miss other options?

4

That’s correct. The tool is for the whole application. – And yes, there is still a manual cleanup for local state required… Your understanding is correct.

It’s the first time that I hear about the requirement to only reset a specific sub-set up stores. Seems not to be a very common pattern, so it requires a none-standard approach.