As of now whenever my kafka streams application restarts it builds the state store from scratch , how do I enabled checkpointing so, that it considers only the missing updates while building the state store
Can you elaborate? Kafka Streams write state do local disk by default. Thus, if should not need to restore anything on restart.
How do you deploy your application? For example, if you are using Kubernetes, you should use stateful sets and persistent volumes to preserve state across restarts.
No, I am not using kubernetes. I have started my application on a VM.
I have come across few applications which have .checkpoint file created in the state store dir and when the application restarts it uses that checkpoint info and restores only the reqquired records in the state store. I just wanted to know from which version of kafka-streams library this is supported.
I have started my application on a VM.
Well, for this case, you also need to ensure that state does not vanish (from my understanding)? If you VM must be build with an attached persistent volume and you would need to configure state.dir to use the persistent volume.
I have come across few applications which have .checkpoint file created in the state store dir and when the application restarts it uses that checkpoint info and restores only the required records in the state store.
Not sure what you mean by this? Btw: the .checkpoint file only contains metadata only. For default RocksDB, the actual data is in the nested rocksdb directory.
I just wanted to know from which version of kafka-streams library this is supported.
It is supported since the original release in Kafka 0.10.0.