Hey everyone,
I started using a Standard Kafka cluster and was surprised by how fast the number of active connections can grow. I started investigating why:
- Every Kafka consumer and producer needs to connect to a leader of every partition it’s consuming/producing to.
- I created a topic with 40 partitions, then checked how many unique leaders were used. In my situation, it was 30 unique leaders.
- In this case, an application that uses 4 Kafka producers creates 120 connections, I was able to confirm it using the Metrics API.
So, a single producer application with a single topic uses more than 10% of the available capacity (the Standard cluster has 1000 active connections count limit). The worst part here: it doesn’t look like there is anything I can do to leverage the number of brokers/leaders because of the “cloud magic” 
Which is very unfortunate, because, for example, when using 5 brokers/unique leaders I’d just end up with 20 active connections…
A few questions I have:
- Am I mistaken by any chance and it’s actually somehow possible to have fewer unique leaders and, therefore, connections?
- Are there any other practices that can help to reduce the number of active connections?
Thank you!
