Im new to the confluent cloud and scraping the metrics with the Prometheus config, which lists each cluster, connector, or ksql db by the ID. If any of those objects are removed for some reason. The scrape will break with a 403 unauthorized error as that specified object no longer exists.
Is there a way to scrape at an ORG level and not be dependent on listing each clusterID.
or if that ID is not available, continue without failing
could you elaborate a bit more?
who did you configure prometheus?
I guess you are using the metrics api?
This is correct, using the metrics api with the generated config
- job_name: Confluent Cloud
thanks for the input.
need to check by myself whether it’s possible.
will come back once tested
same error here
afaik there is no possibility to automate this out of the box
one thing which might work is to get the cluster ids via confluent cli and create the
prometheus.yml by script or similar.
will try this tomorrow
I am facing the same issue.
If one “param” is down, everything is down
And the rate limiting makes it even more complicated. I mean, one could always create multiple scrape jobs ( which would not make much sense) but we would just receive a 429. too many requests.
Like you i am yet to come up with a solution, so i am very eager to hear about your findings.
I was hoping for something like a file based sd, but i cant come up with a way to provide the “resource.kafka.id” part in a flie_sd…
Will follow this thread