KafkaStreams' Sensors

Kafka Streams
1

I want to publish record-e2e-latency p90, p95, p99 to datadog. Apache Kafka. For what I can see KafkaStreams only publishes min, avg, and max by attaching the corresponding Min, Avg, and Max MeasurableStats to the record-e2e-latency sensors in StreamTask.

KafkaStreams.metrics() does publish all metrics. But there is no way to get to the parent Sensor of a Metric. If KafkaStreams also had a sensors() function, I would be able to filter for the record-e2e-latency sensors that I am interested in, and add a Percentiles stat to have KafkaStreams compute the p90, p95, p99 that I need. Alternatively, I could even add a custom made MeasurableStats that published directly to a micrometer Timer, i.e.

  @Override
  public void record(final MetricConfig config, final double value, final long timeMs) {
    e2eMicrometerTimer.record((long) value, TimeUnit.MILLISECONDS);
  }

Another possibility to get to KafkaStreams’ Sensors would be if the MetricsReporter provided them, but similarly to KafkaStreams.metrics(), it only exposes KafkaMetric.

Questions:

  1. is there another possibility to have kafka compute and publish p90, p95 and p99 for record-e2e-latency?
  2. are there any plans to add a method in KafkaStreams to get the list of all Sensors, similar to what streams() does.
  3. are there plans to publish Sensors in MetricsReporter?
  4. are there plans to add a getSensor() property in KafkaMetric?
  5. is there a clever way to get to the record-e2e-latency sensors?

Thanks.

2

I don’t think that there is good way to get what you want right now, and I am not aware of any requests to make these new metric available.

The best way forward would be to file a feature request Jira ticket (Issue Navigator - ASF JIRA) for it, and hope that somebody has interest to work on it and pick it up.

If you want to have this sooner than later, you might want to consider picking up this work yourself and contribute to Kafka (Apache Kafka).