How to check lags for each partition of topics that a consumer group has subscribed to?

Clients
1

Hi Robin, thanks for your blogs and they are really helpful. Actually I have a question on how to check lags for each partition of topics that a consumer group has subscribed. Basically I’m running a Kafka cluster on AWS MSK, and on my Kafka client machine I was able to check the lags by using the Kafka cli:
kafka-consumer-groups --describe --group --bootstrap-server

I’d love to programmatically implement this functionality. Do you have any suggestions on how to achieve that using Golang, or is there any REST API I can call? Thanks and looking forward to your reply!

2

Hi @shaobol!
I don’t have experience with golang clients, but Admin APIs can be used to collect this information: AdminClient has 2 operations that help to calculate consumer group lag:

Comparing offsets stored in Kafka _consumer-offsets topic with the latest offset from a Topic Partition, consumer lag can be calculated.

Here is an example in Java: kafka-connect-restart-by-lag/App.java at 16d2a7c271f297a3c492f478d6c589479f68ffd5 · jeqo/kafka-connect-restart-by-lag · GitHub

There are some tooling to monitor lags that you might find helpful:

HTH,
Jorge.

2 Likes