Best Practice for Configuring Kafka 3.6.1 Controllers and Brokers

Kafka Connect Self-Managed Connectors
1

I am setting up a Kafka 3.6.1 cluster with three nodes and have created two separate systemd service files—one for the Kafka controller and another for the broker. The controller is running on port 9093, and the broker is also configured to use 9092.

Currently, I am using controller.properties for the controllers and server.properties for the brokers. However, I noticed that Kafka 3.6.1 (and later versions) allow using a single server.properties file for both roles by specifying process.roles=controller or process.roles=broker.

Questions:

  1. What is the industry best practice for configuring controllers and brokers in Kafka 3.6.1+?
  2. Should I continue using separate properties files, or is it recommended to merge everything into a single server.properties file?
  3. Are there any advantages or disadvantages to either approach in terms of manageability, scalability, or performance?
2

hey @ne2213

for a larger prod deployment it’s recommended to split broker and controller to separate nodes.
therefore I would recommend to split the controller and broker properties file to make it easier to switch to separate nodes if you need to do so.

3

Hi @mmuehlbeyer , thanks for your response.
I am running both the Kafka broker and Kafka controller services on each of my three servers. Would it be recommended to use separate nodes/VMs for the Kafka controllers, or does it depend on the traffic?

4

Hi @ne2213
depends on the traffic.
I’ve seen some installations in production where broker and controllers running on the same node.
though as said there might be the need to split the roles to different nodes and therefore separating the configs upfront would be highly recommended.

another point to consider is if you’re loosing a node to whatever reason you will loose a controller and a broker :wink: