Need help to pull KSQLDB pull metrics in prometheus

Hi All,

I am trying to pull the KSQLDB Pull queries and User-defined functions in prometheus using JMX exporter.
For this I have tried enabling ksql.query.pull.metrics.enabled=true in ksql-server.properties.
Also, added few entries in ksql_rule.yml file as below:

# io.confluent.ksql.metrics:type=_confluent-ksql-pull-query
  - pattern: io.confluent.ksql.metrics<type=_confluent-ksql-pull-query><>([^:]+)
    name: "ksql_ksql_pull_query_stats_$2"
    labels:
      ksql_query: $1

But still, unable to fetch the metrics in prometheus.

Please help me with this to pull the required data in prometheus.

Hi,

Prometheus with any errors in the logfile?
Did you check whether ksqldb JMX Port is accessible (via curl or similar)

I didn’t found any errors in prometheus.
JMX port is accessible, as I am publishing all other metrics using JMX exporter.
I think there is some problem with the ksql_rule.yml

I am adding a rule like below:

io.confluent.ksql.metrics:type=_confluent-ksql-pull-query

  • pattern: io.confluent.ksql.metrics<type=confluent-ksql-pull-query><>([^:]+)
    name: "ksql_ksql_pull_query_stats
    $2"
    labels:
    ksql_query: $1

Please help if there is some problem with the rule file.

@Rajesh as an FYI, there was a conversation about this in the Confluent Community Slack group here: https://confluentcommunity.slack.com/archives/C6UJNMY67/p1620051507335500 I’m not sure if you are signed up there, but maybe the individuals there can help you diagnose.

I am able to fetch the ksql pull metrics using JMX exporter as below.

io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-total (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-remote-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-latency-max (Double) = nan
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-response-size (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-error-total (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-remote-count (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-error-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-local-count (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-distribution-90 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-distribution-50 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-distribution-75 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-request-size (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-local-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-latency-min (Double) = nan
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-distribution-99 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-latency-avg (Double) = nan
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-total (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-remote-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-latency-max (Double) = nan
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-response-size (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-error-total (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-remote-count (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-error-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-local-count (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-distribution-90 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-distribution-50 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-distribution-75 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-request-size (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-local-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-latency-min (Double) = nan
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-distribution-99 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-latency-avg (Double) = nan

Now, I think I am not able to parse this values in prometheus.
Not sure, but please suggest if any changes in rule files is required.

Below is the rule defined in ksql_rule.yml

# io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=
 # io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-total
 # _confluent-pull-query,ksql_service_id=poc_/
 # pattern: io.confluent.ksql.metrics:type=_confluent-pull-query
 - pattern: io.confluent.ksql.metrics<type=_confluent-pull-query,ksql_service_id=poc/*>
   name: "ksql_confluent-pull-query_query_stats_$2"
   labels:
     ksql_confluent-pull_query: $1

This topic was automatically closed after 30 days. New replies are no longer allowed.