Using a String Key in Messages - Excessive Increase in Resource Usage

[a.j.g]

Hi all,

I recently started using message keys to partition a topic for the first time in our cluster but it has caused a massive increase in broker CPU and network usage - CPU usage is now ~2.2x what it was before, network bytes in and out are both ~1.6x, network packets in is 3x (!) and network packets out is ~2.1x what it was before across all 3 brokers uniformly. The keyed messages are much smaller in both size and quantity than just a single other non-keyed topic so these increases are definitely unexpected. Does anyone have any insight into what’s happening? Is this expected behaviour based on how keys work or do I maybe have some misconfiguration? Brokers are Kafka 2.6 and producer is php-rdkafka 6.0 on top of librdkafka 1.5.2

I changed the messages to be without the key only so that I could visualize how much of the resource usage was from the key, here’s the screenshots. First circle is the start of sending messages with key + value, second circle is the start of sending messages with value only.

Thanks in advance!

[a.j.g]

[a.j.g]

For anyone that comes across this, for me this issue was being caused by a too-low linger.ms and, for some reason, a high batch.size. Changing linger.ms from 500 to 2000 and batch.size from 500,000 to default has resulted in much better resource utilization on the cluster.