In 2015 there was a question in stackoverflow about dynamically changing the number of partitions in a topic. After all these years and with the replacement zookeeper with KRaft, is it possible currently to dynamically change the number of partitions in a topic with no downtime? (This is for the case of needing to increase paralelism-performance in the consumer side.)
Can you provide what you have tried so far? This command seems like it exists already?
kafka-topics --topic X --alter --partitions N
Help output
--partitions <Integer: # of partitions> The number of partitions for the topic
being created or altered (WARNING:
If partitions are increased for a
topic that has a key, the partition
logic or ordering of the messages
will be affected). If not supplied
for create, defaults to the cluster
default.
I have not tried this on run-time. Was wondering what would be the consequence of increasing the number of partitions while the topic is being used. But I can try it.
Thank you.
Kafka clients refresh metadata every 5 minutes on their own. In the meantime, existing data will not be migrated when you increase partition count, and you will lose ordering on newly produced messages for any matching keys from previous records/partitions.