Multi-tenant Kafka platform sizing

Architecture and Design
1

how do we calculate number of brokers and zookeepers required for a multi-tenant platform. Also number of partitions across brokers, replication factor, in-sync replica. Is there any calculation to do that

2

Hi Lisha,

did you consider trying

Could be a good point to start with.

Best,
Michael

3

Thanks Michael. That was helpful.

I am also looking for some guidance on how to design a multi-tenant (shared) platform.
I have 6 brokers , as per the eventsizer Maximum topic partitions: 14,400 (includes replica partitions). Is this the maximum partitions allowed in the broker combining all the partitions?
What if the max partition limit is reached? what will be the behavior.
Is horizontal scaling recommended?

4

ok cool

it’s recommend to have not more than 4000 partitions per broker nevertheless it’s no
hard limit.
thiss is related to huge partition reassignments in case of a node failure or similar.

cluster wide you could have 200000 partitions see

5

Hi Michael

Thanks for the clarification.

We want to build an event backbone. We are currently looking to have a kafka cluster with 6 brokers, but eventually we would like to scale up and add more brokers to the clusters. We want this setup to use payments data and hence a critical service.
Do you have any guidance on how to scale up without any data loss. We will be using Community edition.

Thanks
Lisha