Solution to emit consumers lag into Prometheus

Kafka Streams
1

Hi,
Solutions can include:

  • a KafkaStreams job that works with __offsets and consumer_offsets as KTable
  • Burrow
  • Kafka Lag Exporter
  • Minion
  • a script with CLI that gets the top offsets of partitions and the groups offsets on them
  • from the consumer (I am excluding this, due to the fragility - we want lag visible when cluster is down)

Do you have any experience with any consumer-independent solution of tracking lag and emitting into a DataDog / Prometheus, etc?

2

@nmarasoiu Check out this recent blog from @awalther28: Monitoring Your Event Streams: Tutorial for Observability Into Apache Kafka Clients