Issue displaying metrics in Prometheus from Confluent Cloud

Confluent Cloud
1

Hi all,

I cannot retrieve any metrics from confluent cloud in Prometheus, in the table tab. But in the graph tab, I can successfully see them.

In particular, I’d like to retrieve the metric: “confluent_kafka_server_consumer_lag_offsets”

in the table tab: No data queried yet
In the graph tab: I see datas

I can make query from the table tab work and make an instant query by using: confluent_kafka_server_consumer_lag_offsets[10m] or last_over_time(confluent_kafka_server_consumer_lag_offsets[10m]

Below is my Prometheus configuration:

- job_name:  Confluent Cloud
  honor_timestamps: true
  track_timestamps_staleness: true # this is me trying to fix my current issue using this parameter. Without it, I also have the initial issue
  params:
    resource.kafka.id:
    - consumer1
    - consumer2
    - consumer3
  scrape_interval: 1m
  scrape_timeout: 1m
  scrape_protocols:
  - OpenMetricsText1.0.0
  - OpenMetricsText0.0.1
  - PrometheusText0.0.4
  metrics_path: /v2/metrics/cloud/export
  scheme: https
  enable_compression: true
  basic_auth:
    username: xxxx
    password: <secret>
  follow_redirects: true
  enable_http2: true
  http_headers: null
  static_configs:
  - targets:
    - xxx.xxx.confluent.cloud # hidden

My question is the following: Is it expected to not receive any datas in the table tab section? (instant query)

If it is not, could you please advise, thank you!

Feel free to ask questions if needed.

Additional information:

Prometheus version: 2.53.1
Prometheus is running on GKE (Kubernetes)

2

hi @ptran32

qq: are others metrics showing up in table tab?
for me it worked after switching to “classical ui” and then back to “new ui”

best,
michael

3

Hi Michael,

All confluent_kafka* metrics returns nothing from the “table” tab, but like the confluent_kafka_server_consumer_lag_offsets they all works when adding a [10m] in the query or searching in “graph” metrics.

I’d like to add that the configuration above is not exact, especially for scrape_interval & scrape_timeout that is not 1m but actually 5m (I am editing the above code). We cannot have a 1m scrape_interval because we have 2 Prometheus pods and we would reach the limit of 80 query per hour

4

I don’t have any option from my Prometheus UI to switch to any “classical” or “new” UI by the way

5

hmm ok I see

did you take a look at

6

Thanks for sharing. This documentation is pretty much what we are currently using, but with Prometheus configuration.

Do you know if there’s any steps/guide I could follow to make the metrics appearing?

Thank you

1 Like
7

I have the same question. We use 7 minutes but it is too much :neutral_face: