I am experiencing intermittent data retrieval issues using kcat against a Kafka cluster hosted on AKS. The command sometimes hangs without returning data or errors. Debug logs show metadata errors like:
ListOffsetsRequest failed: Broker: Not leader for partition
Offset reply error for topic [partition]: Broker: Not leader for partition
We have verified AKS networking and Kafka service configurations are correct. The problem seems related to the kcat command targeting only one broker, which isn’t always the partition leader.
Could someone please advise how to properly configure kcat or Kafka to avoid these silent hangs? Any insights or suggestions on troubleshooting this metadata/leadership issue would be greatly appreciated.
Thank you in advance!