Unable accessing Kafka brokers from outside the cluster

Clients
1

I am experiencing intermittent data retrieval issues using kcat against a Kafka cluster hosted on AKS. The command sometimes hangs without returning data or errors. Debug logs show metadata errors like:

ListOffsetsRequest failed: Broker: Not leader for partition
Offset reply error for topic [partition]: Broker: Not leader for partition

We have verified AKS networking and Kafka service configurations are correct. The problem seems related to the kcat command targeting only one broker, which isn’t always the partition leader.

Could someone please advise how to properly configure kcat or Kafka to avoid these silent hangs? Any insights or suggestions on troubleshooting this metadata/leadership issue would be greatly appreciated.

Thank you in advance!

2

How did you configure Kafka’s advertised listeners and set up networking for external clients? I am thinking that the problem is in this area rather than kcat, but it would also be worthwhile to try a different client just to rule out anything kcat-specific.

3

Hi @dtroiano,

Hey, thanks for your response.

We have configured Kafka following this documentation:

The internal access is working perfectly, and we can consistently fetch data from all brokers.

However, for external access, Kafka only sometimes returns data from one broker, even though we have three brokers in the cluster.

Do you have any idea what might be causing this issue?

4

we are also facing similar problem , @dtroiano Dave can you pls bring your specialist insight on this problem.