We are a small enterprise and recently deployed a microservice-based application that uses Apache Kafka. We are now preparing to move it into production, and I have a few questions about how we should properly deploy Kafka in KRaft mode:
Initial setup:
Should we start with a single node (controller + broker) for production, or is it better to set up multiple nodes from the beginning? in case we do this how to scale up kafka when there is a lot of load
Multiple brokers on one server:
Is it a good practice to run multiple brokers on a single server (with different ports) and assign multiple partitions this way, or should each broker always run on a separate machine/VM?
I’d really appreciate best practices or lessons learned from others who have deployed Kafka in production under similar constraints.
from a performance and resilience perspective it’s better to have at least 3 brokers and controllers
I would highly recommend to have each broker on a separate machine/VM
you could think about having controller and brokers on the same machine for the beginning, though as said keep the brokers on separate nodes.
and not to forget: moving controller and/or brokers to other nodes during the workload needs to be planned carefully to keep the downtime small (or omit completly)
