Active Connections Count in Confluent Cloud

Confluent Cloud
1

Hey everyone,

I started using a Standard Kafka cluster and was surprised by how fast the number of active connections can grow. I started investigating why:

  • Every Kafka consumer and producer needs to connect to a leader of every partition it’s consuming/producing to.
  • I created a topic with 40 partitions, then checked how many unique leaders were used. In my situation, it was 30 unique leaders.
  • In this case, an application that uses 4 Kafka producers creates 120 connections, I was able to confirm it using the Metrics API.

So, a single producer application with a single topic uses more than 10% of the available capacity (the Standard cluster has 1000 active connections count limit). The worst part here: it doesn’t look like there is anything I can do to leverage the number of brokers/leaders because of the “cloud magic” :slightly_smiling_face: Which is very unfortunate, because, for example, when using 5 brokers/unique leaders I’d just end up with 20 active connections…

A few questions I have:

  • Am I mistaken by any chance and it’s actually somehow possible to have fewer unique leaders and, therefore, connections?
  • Are there any other practices that can help to reduce the number of active connections?

Thank you!

2

Hello Yaroslav Tkachenko,

Did you ever figure this out as far as the active connections?

We are running into a similar situation.

Thank you.

3

Same issue as above, up :arrow_up: :pray:

4

Good news: the limit is 10 000 active connections for standard cluster now :smile: