Streams application memory consumption keeps on increasing

Kafka Streams
1

I have an kafka-streams application with state store in it. I have a load running which keeps writings unique keys into the state store . I see that the memory usage keeps on increasing and eventually I am getting OutOfMemoryException. I am running the streams application with default rocksDB configuration.

How can I stop the increasing memory consumption?

2

hey,

did you consider
https://docs.confluent.io/platform/current/streams/developer-guide/memory-mgmt.html#kstreams-memory-management

3

Thanks for the reply.

I think the index cache and filter block are causing the issue. I just want to know how can I limit the index cache and filter block memory consumption without making it a part of the block cache which is used for data blocks.

4

which keeps writings unique keys into the state store

Well, it’s obvious that you run out of memory/disk eventually. A state store will keep every record for this pattern. It’s like a database: if you don’t delete anything, it just keeps growing. A state store is not an “cache” and there is not time-to-live, ie, no expiration of “old” data.

To configure RocksDB itself, you can implement RocksDBConfigSetter and pass it into the configuration: Configuring a Streams Application | Confluent Documentation

This blog post might help: Performance Tuning RocksDB for Kafka Streams’ State Stores

5

I have gone through theabove blog post . It mentions using the options
options.setMaxWriteBufferNumber() & options.setWriteBufferSize(); to limit the memory. But my issue is related to index cache and filter block because memtables by default are only 3 and each has a size of 16MB. Also, block cache is bounded to 10MB So, there is possibility that index cache and filter data might be consuming the heap space.

  1. In some blog post I have seen by setting the option “max_open_files” in rocksDB can limit the number of open files . So, I just wanted to know is this the right way because its default value is -1 which means all files are open.
  2. Also, should I increase the default size of SST file from 66MB to 132MB ?