Problem Statement
We have a Kafka publishing API, written in Golang, which handles publishing requests to multiple Kafka topics. The API is designed to run with 5 worker goroutines, each maintaining its own Kafka writer instance to efficiently batch and push messages.
To implement key-based partitioning, we plan to use Kafka’s built-in balancer:
Balancer: &kafka.Hash{}
This balances messages across partitions by hashing the message key.
However, the official documentation for the Balancer interface includes the following note:
“An application should refrain from using a balancer to manage multiple sets of partitions (from different topics for example). Use one balancer instance for each partition set, so the balancer can detect when the partitions change and assume that the Kafka topic has been rebalanced.”
This raises a concern: our current setup uses a shared set of 5 writers across all topics, which implies that a single balancer instance may end up managing multiple partition sets (i.e., different topics). This could lead to incorrect partition assignments or ineffective balancing when Kafka topics undergo partition changes or rebalancing.
Kafka Version : 3.6.1
Go Library : GitHub - segmentio/kafka-go: Kafka library in Go
Objective:
We need to determine an efficient and scalable way to:
-
Ensure correct partitioning behavior per topic,
-
Avoid violating the balancer interface contract,
-
Maintain performance and resource efficiency in a multi-topic, multi-writer architecture.