Hello,
We are about to shift from an on-prem cp deployment to Confluent Cloud.
While on prem we used metricbeat and jolokia to monitor metrics.
Now we would like to do something of the same, and when reaching out to Elastic we were told that the Metricbeat Kafka module could be used to monitor the partition and consumer group metrricsets.
Following the metricbeat kafka module documentation i have set up the required ACL’s.
But when starting metricbeat i am getting the following error,
“error in connect: getting cluster client for advertised broker with address pkc-w7d6j.germanywestcentral.azure.confluent.cloud:9092: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)”
I can reach the cluster using telnet, so i don’t think this is a network/firewall issue, but i could be wrong.
Does anyone here know if what i am trying to achieve is even possible? and if it is, how i would go about it?
Best regards
Oelsner
hi @Oelsner
could you provide some more details (or a link) how you configureed metribeat?
did you create a service account and an api key?
best,
michael
Hi Michael,
Sorry about the latency, weekend for the better of me 
Yes i did create a service account called metricbeatSvc and an API key.
I am using the key as username and the secret as password.
- module: kafka
metricsets:
- partition
- consumergroup
period: 10s
hosts: ["pkc-w7d6j.germanywestcentral.azure.confluent.cloud:9092"]
username: "Key"
password: "Secret"
Please let me know if there is more info i can provide.
Best regards
Oelsner
hi @Oelsner
thanks for the info.
did you check whether the login works with confluent cli?
if yes could you provide some details how metricbeat try to accomplish the login and so on?
best
michael
Hi Michael,
The login works flawlessly using Confluent CLI, and i am consuming messages using it as i type this.
All the metricbeat beat documentation mentions is that it us using the kafka protocol, which i suspect is tcp.
I am also in a dialog with Elastic support who tells me that it is a generic message they throw when there is any kind of connection or authentication issues talking to kafka.
Can i provide more info?
Best regards
Oelsner
Hi @Oelsner
hmm strange, one thing worth to try:
did you check the
sasl.mechanism
parameter?
best,
michael
Hi @mmuehlbeyer
Sorry about the latency, my attention was directed somewhere else.
The sasl.mechanism used is PLAIN, as i did when running it on confluent platform.
I have sort of given up on it or semi concluded that it wont work
However i would not mind being corrected
Thanks for the replies though.
Best regards
Oelsner
Hi @Oelsner
no worries
it may be worth to try using the cloud metrics endpoint provided by confluent
https://api.telemetry.confluent.cloud/
and scrape it with the metricbeat http module
see Confluent Cloud Metrics | Confluent Documentation for details
best,
michael
Hi @mmuehlbeyer
Thanks for the advice, i will try and give it try one of the coming days and report back here with my findings
Cheers
Oelsner
1 Like
Better late than never
All running in kubernetes.
We ended up using Prometheus scraping the exports endpint.
Using Prometheus remote_write to ship the metrics to an elastic agent.
Now all metrics are perfectly visible in Kibana
Best
Christian Oelsner
1 Like