Lag metric to prometheus

Confluent Cloud
1

What are my options for exporting the max lag of a topic into promethus?

I have reviewed the metrics API already and don’t see anything at the consumer group level.

Seems like there should be a jmx exporter endpoint that we could consumer from.

Thanks
Zach

2

Have you checked GitHub - seglo/kafka-lag-exporter: Monitor Kafka Consumer Group Latency with Kafka Lag Exporter? It’s probably the easiest way to export the consumer lag from a Confluent Cloud deployment to Prometheus.

1 Like
3

Hi @zachariahyoung, per this doc at the moment this isn’t possible using the metrics API.

@mcascallares’s suggestion is a good alternative.

4

@mcascallares Yes we have reviewed this tool and have it working.

It just seems strange that I would have to use this tool.

Why can’t I pull this out of a jmx exporter or the Metrics API.

5

@mcascallares @zachariahyoung
Can someone please share here how we can pass confluent metrics api endpoints in this lag exporter?(GitHub - seglo/kafka-lag-exporter: Monitor Kafka Consumer Group Latency with Kafka Lag Exporter) …
or how we can connect lag exporter to our confluent cloud endpoint? I couldn’t see this in the documentation.

6

@rmoff could you please help here?

7

@vaseemt

kafka-lag-exporter does not use the Confluent Metrics endpoints, it is configured as if it were a kafka client it uses the Kafka API

For example configurations check:

Here is an example of a what you need to set to connect to Confluent Cloud

bootstrap.servers=pkc-xxxxx.us-east-1.aws.confluent.cloud:9092
ssl.endpoint.identification.algorithm=https
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="{{ CLUSTER_API_KEY }}" password="{{ CLUSTER_API_SECRET }}";

This post was created on 2 years ago when Confluent Cloud did not have Consumer lag in its metrics the metric is now directly available see the documentation for more reference:
https://docs.confluent.io/cloud/current/monitoring/monitor-lag.html#monitor-consumer-lag-via-the-metrics-api