Confluent Cloud error: Broker may not be available

[Suja]

Hi @rmoff

I am facing similar issue.

Our java application connects to confluent cloud kafka - azure to create kafka topics, etc. There are two logs that flows continuosly in our java app, although connection to cluster and creation of topics are successful.

{“message”:“[AdminClient clientId=adminclient-1] Connection to node -1 (pqc-abcny.eastus.azure.confluent.cloud/x.x.x.x:9092) could not be established. Broker may not be available.”,“timestamp”:“2021-11-22T06:38:53.794270Z”,“fields”:{“processId”:“6785@HONC02G93RUMD6M”,“level”:“WARNING”}}

{“message”:“[AdminClient clientId=adminclient-1] Metadata update failed”,“timestamp”:“2021-11-22T06:39:23.806013Z”,“fields”:{“processId”:“6785@HONC02G93RUMD6M”,“level”:“INFO”,“exception”:{“cause”:“Call(callName=fetchMetadata, deadlineMs=1637563163795, tries=1, nextAllowedTryMs=1637563163901) timed out at 1637563163801 after 1 attempt(s)”,“stackTrace”:“org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1637563163795, tries=1, nextAllowedTryMs=1637563163901) timed out at 1637563163801 after 1 attempt(s)\nCaused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata\n”}}}

Thanks in advance!

[mmuehlbeyer]

Hi @Suja

in general it might be related to the advertised listener parameter in server.properties.
As you are using confluent cloud it should be already configured correctly I guess.

Nevertheless some further reading

Running Kafka Connect (Sink) On Separate Machine From Zookeeper & Topic (non localhost)

Greetings, Beginner Problem: I’m running a Kafka environment (machine #1) on a standalone machine and MySQL on a separate machine (machine #2). The Kafka environment is running well, with an Avro(ized) topic. My goal is to use a Connector to sink the data from the topic to the MySQL database. However, when I attempt to start up the Connect instance on the database machine (#2) -pointed to the zookeeper socket of the Kafka environment (#1) -, I receive the following message: WARN [AdminClien…

Why Can’t I Connect to Kafka? | Troubleshoot Connectivity

HTH,
Michael

[Suja]

Hi @mmuehlbeyer

Yes, we are using confluent cloud. Is there a way to verify the properties using api or ccloud cli.

Thanks!

[mmuehlbeyer]

Hi @Suja

you could try with
confluent kafka broker describe [id] [flags]

or
confluent kafka cluster describe [id] [flags]

maybe you need to upgrade you confluent cli
https://docs.confluent.io/confluent-cli/current/migrate.html

[Suja]

Hi @mmuehlbeyer

I checked with the latest confluent cli. I was not able to find the advertised listener parameter. Below are the available properties. Is there any other way to debug this issue.

| Id
| Name
| Type
| Ingress
| Egress
| Storage
| Provider
| Availability
| Region
| Status
| Endpoint
| RestEndpoint

Thanks!

[mmuehlbeyer]

Hi Suja,

need to check by myself.
Are you using spring boot?
Could you check the following setting:

retry.backoff.ms

Best,
Michael

[Suja]

Hi @mmuehlbeyer

Yes, using spring boot.
retry.backoff.ms = 100

Thanks

[mmuehlbeyer]

Hi Suja,

thanks, looks good my recommendation would have been to set the parameter to 100.

Does the error also occur if you try with for example kafka-console-producer?

[Suja]

Hi @mmuehlbeyer

I am using confluent cloud AdminClient only to manage topics and acls.

[mmuehlbeyer]

Hi @Suja

ok I see
what happens if you increase the mentioned parameter to let’s say 200?

could share your producer config?

[Suja]

Hi @mmuehlbeyer

I think all those are managed by confleunt. I dont see apis/cli to update configs.

AdminClientConfig values:                                                                         
bootstrap.servers = [pkc-epwny.eastus.azure.confluent.cloud:9092]                                                                         
client.dns.lookup = use_all_dns_ips                                                                         
client.id =                                                                          
connections.max.idle.ms = 300000                                                                         
default.api.timeout.ms = 60000                                                                         
metadata.max.age.ms = 300000                                                                         
metric.reporters = []                                                                         
metrics.num.samples = 2                                                                         
metrics.recording.level = INFO                                                                         
metrics.sample.window.ms = 30000                                                                         
receive.buffer.bytes = 65536                                                                         
reconnect.backoff.max.ms = 1000                                                                        
reconnect.backoff.ms = 50                                    
request.timeout.ms = 30000 
retries = 2147483647                                                                         
retry.backoff.ms = 100                                                                         
sasl.client.callback.handler.class = null                                                                         
sasl.jaas.config = [hidden]                                                                         
sasl.kerberos.kinit.cmd = /usr/bin/kinit                                                                         
sasl.kerberos.min.time.before.relogin = 60000                                                                         
sasl.kerberos.service.name = null                                                                         
sasl.kerberos.ticket.renew.jitter = 0.05                                                                         
sasl.kerberos.ticket.renew.window.factor = 0.8                                                                         
sasl.login.callback.handler.class = null                                                                        
sasl.login.class = null                                                                         
sasl.login.refresh.buffer.seconds = 300                                                                         
sasl.login.refresh.min.period.seconds = 60                                                                         
sasl.login.refresh.window.factor = 0.8                                                                         
sasl.login.refresh.window.jitter = 0.05                                                                         
sasl.mechanism = PLAIN                                                                         
security.protocol = SASL_SSL                                                                         
security.providers = null                                                                        
send.buffer.bytes = 131072                                                                         
socket.connection.setup.timeout.max.ms = 30000                                                                         
socket.connection.setup.timeout.ms = 10000                                                                        
ssl.cipher.suites = null                                                                         
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]                                                                      
ssl.endpoint.identification.algorithm = https                                                                        
ssl.engine.factory.class = null                                                                         
ssl.key.password = null                                                                         
ssl.keymanager.algorithm = SunX509                                                                      
ssl.keystore.certificate.chain = null                                                                         
ssl.keystore.key = null                                                                         
ssl.keystore.location = null                                                                         
ssl.keystore.password = null                                                                         
ssl.keystore.type = JKS                                                                       
ssl.protocol = TLSv1.3                                                                         
ssl.provider = null                                                                        
ssl.secure.random.implementation = null                                                                         
ssl.trustmanager.algorithm = PKIX                                                                         
ssl.truststore.certificates = null                                                                        
ssl.truststore.location = null                                                                         
ssl.truststore.password = null                                                                        
ssl.truststore.type = JKS

[prkashyap]

Hi,
I’m facing same issue. Were you able to solve? Could you please share the solution?

Thanks and regards