Confluent Cloud metrics using Metricbeat

Hello,
We are about to shift from an on-prem cp deployment to Confluent Cloud.
While on prem we used metricbeat and jolokia to monitor metrics.
Now we would like to do something of the same, and when reaching out to Elastic we were told that the Metricbeat Kafka module could be used to monitor the partition and consumer group metrricsets.
Following the metricbeat kafka module documentation i have set up the required ACL’s.
But when starting metricbeat i am getting the following error,

error in connect: getting cluster client for advertised broker with address pkc-w7d6j.germanywestcentral.azure.confluent.cloud:9092: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

I can reach the cluster using telnet, so i don’t think this is a network/firewall issue, but i could be wrong.

Does anyone here know if what i am trying to achieve is even possible? and if it is, how i would go about it?

Best regards
Oelsner

hi @Oelsner

could you provide some more details (or a link) how you configureed metribeat?

did you create a service account and an api key?

best,
michael

Hi Michael,

Sorry about the latency, weekend for the better of me :slight_smile:
Yes i did create a service account called metricbeatSvc and an API key.

I am using the key as username and the secret as password.

- module: kafka                                                                                                                                                                                                                                
  metricsets:                                                                                                                                                                                                                                    
    - partition                                                                                                                                                                                                                                  
    - consumergroup
period: 10s
hosts: ["pkc-w7d6j.germanywestcentral.azure.confluent.cloud:9092"] 

username: "Key"
password: "Secret"


Please let me know if there is more info i can provide.

Best regards
Oelsner

hi @Oelsner

thanks for the info.
did you check whether the login works with confluent cli?
if yes could you provide some details how metricbeat try to accomplish the login and so on?

best
michael

Hi Michael,
The login works flawlessly using Confluent CLI, and i am consuming messages using it as i type this.
All the metricbeat beat documentation mentions is that it us using the kafka protocol, which i suspect is tcp.
I am also in a dialog with Elastic support who tells me that it is a generic message they throw when there is any kind of connection or authentication issues talking to kafka.

Can i provide more info?

Best regards
Oelsner

Hi @Oelsner

hmm strange, one thing worth to try:

did you check the

sasl.mechanism

parameter?

best,
michael

Hi @mmuehlbeyer
Sorry about the latency, my attention was directed somewhere else.
The sasl.mechanism used is PLAIN, as i did when running it on confluent platform.

I have sort of given up on it or semi concluded that it wont work :slight_smile:
However i would not mind being corrected :slight_smile:

Thanks for the replies though.

Best regards

Oelsner

Hi @Oelsner

no worries :slight_smile:

it may be worth to try using the cloud metrics endpoint provided by confluent
https://api.telemetry.confluent.cloud/
and scrape it with the metricbeat http module

see Confluent Cloud Metrics | Confluent Documentation for details

best,
michael

Hi @mmuehlbeyer
Thanks for the advice, i will try and give it try one of the coming days and report back here with my findings :slight_smile:

Cheers
Oelsner

1 Like