Configuring Kafka Streams Internal Topics

Kafka Streams
1

Is it possible to configure Kafka Streams Internal topics (changelog & repartition)?
Setting the number of parititions, replication factor, and retention values?
Thanks

2

You can use topic. prefix on the configs:

There is also Materialized object that allows you to control on a per store/changelog level: Materialized (streams 3.6.1 API)

3

Thanks. This is just what I was looking for .
Is the number of partitions for changelog & repartions topics controlled by the input topic, and something that can not be configured?

4

Yes. If you want to change it, you would need to add an explicit .repartition() operation which allows you to set the number of partitions for the repartition topic (the downstream sub-topology, will get the same number of tasks, and thus the same number of changelog topic partitions).

The number of changelog topic partitions cannot be set – it’s always coupled to the number of tasks, which is derived from the number of input/repartition topic partitions of the sub-topology.

1 Like
5

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.