Adding new broker to cluster

Ops
1

Hello,
We have 3 node broker and 3 node zookeeper cluster running on kubernetes. We use helm charts for deployment.
Currently we have 4700 partitions on each broker and now we are planning to add another broker to get more partitions. We have enough storage on brokers and don’t see any performance issues apart from high CPU usage sometimes.

My question is there any parameter we can decide when to another broker or any limitation on number of partitions per broker?

2

hey @naveen8384

there is a generic recommendation to have not more than 4000 partitions per broker and not more than 200k partitions per cluster.

More partitions might lead to longer downtimes or performance issues (as you may already noticed)

Here are some blog post and links around that topic:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-578%3A+Add+configuration+to+limit+number+of+partitions

hth,
michael

3 Likes