Need help to pull KSQLDB pull metrics in prometheus

ksqlDB
1

Hi All,

I am trying to pull the KSQLDB Pull queries and User-defined functions in prometheus using JMX exporter.
For this I have tried enabling ksql.query.pull.metrics.enabled=true in ksql-server.properties.
Also, added few entries in ksql_rule.yml file as below:

# io.confluent.ksql.metrics:type=_confluent-ksql-pull-query
  - pattern: io.confluent.ksql.metrics<type=_confluent-ksql-pull-query><>([^:]+)
    name: "ksql_ksql_pull_query_stats_$2"
    labels:
      ksql_query: $1

But still, unable to fetch the metrics in prometheus.

Please help me with this to pull the required data in prometheus.

2

Hi,

Prometheus with any errors in the logfile?
Did you check whether ksqldb JMX Port is accessible (via curl or similar)

3

I didn’t found any errors in prometheus.
JMX port is accessible, as I am publishing all other metrics using JMX exporter.
I think there is some problem with the ksql_rule.yml

I am adding a rule like below:

io.confluent.ksql.metrics:type=_confluent-ksql-pull-query

  • pattern: io.confluent.ksql.metrics<type=confluent-ksql-pull-query><>([^:]+)
    name: "ksql_ksql_pull_query_stats    $2"
    labels:
    ksql_query: $1

Please help if there is some problem with the rule file.

4

@Rajesh as an FYI, there was a conversation about this in the Confluent Community Slack group here: Slack I’m not sure if you are signed up there, but maybe the individuals there can help you diagnose.

5

I am able to fetch the ksql pull metrics using JMX exporter as below.

io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-total (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-remote-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-latency-max (Double) = nan
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-response-size (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-error-total (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-remote-count (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-error-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-local-count (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-distribution-90 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-distribution-50 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-distribution-75 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-request-size (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-local-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-latency-min (Double) = nan
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-distribution-99 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=poc_/pull-query-requests-latency-avg (Double) = nan
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-total (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-remote-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-latency-max (Double) = nan
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-response-size (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-error-total (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-remote-count (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-error-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-local-count (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-distribution-90 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-distribution-50 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-distribution-75 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-request-size (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-local-rate (Double) = 0.0
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-latency-min (Double) = nan
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-distribution-99 (Double) = nan
io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-latency-avg (Double) = nan

Now, I think I am not able to parse this values in prometheus.
Not sure, but please suggest if any changes in rule files is required.

Below is the rule defined in ksql_rule.yml

# io.confluent.ksql.metrics:type=_confluent-pull-query,ksql_service_id=
 # io.confluent.ksql.metrics:type=_confluent-ksql-poc_pull-query/pull-query-requests-total
 # _confluent-pull-query,ksql_service_id=poc_/
 # pattern: io.confluent.ksql.metrics:type=_confluent-pull-query
 - pattern: io.confluent.ksql.metrics<type=_confluent-pull-query,ksql_service_id=poc/*>
   name: "ksql_confluent-pull-query_query_stats_$2"
   labels:
     ksql_confluent-pull_query: $1
6

This topic was automatically closed after 30 days. New replies are no longer allowed.