Kafka Broker failed to connect to Zookeeper

Hi All,

We are using confluent helm charts, which are using zookeeper image version 7.6.1 and kafka image version 7.6.1.

this are the logs we got in broker pod.

===> User
uid=0(root) gid=0(root) groups=0(root)
===> Configuring …
Running in Zookeeper mode…
SASL is enabled.
===> Running preflight checks …
===> Check if /var/lib/kafka/data is writable …
===> Check if Zookeeper is healthy …
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/share/java/kafka/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/kafka/kafka-vault-loader.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/cp-base-new/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See SLF4J Error Codes for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
[2024-09-05 06:20:16,545] INFO SASL is enabled. java.security.auth.login.config=file:/tmp/kafka-server-jaas.conf (io.confluent.admin.utils.ClusterStatus)
[2024-09-05 06:20:16,574] INFO Client environment:zookeeper.version=3.8.3-6ad6d364c7c0bcf0de452d54ebefa3058098ab56, built on 2023-10-05 10:34 UTC (org.apache.zookeeper.ZooKeeper)
[2024-09-05 06:20:16,574] INFO Client environment:host.name=kafka-0.kafka-svc.data-platform.svc.cluster.local (org.apache.zookeeper.ZooKeeper)
[2024-09-05 06:20:16,574] INFO Client environment:java.version=11.0.21 (org.apache.zookeeper.ZooKeeper)
[2024-09-05 06:20:16,574] INFO Client environment:java.vendor=Azul Systems, Inc. (org.apache.zookeeper.ZooKeeper)
[2024-09-05 06:20:16,574] INFO Client environment:java.home=/usr/lib/jvm/java-11-zulu-openjdk-ca (org.apache.zookeeper.ZooKeeper)
[2024-09-05 06:20:16,580] INFO Initiating client connection, connectString=zk-svc:2182 sessionTimeout=40000 watcher=io.confluent.admin.utils.ZookeeperConnectionWatcher@38c5cc4c (org.apache.zookeeper.ZooKeeper)
[2024-09-05 06:20:16,587] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2024-09-05 06:20:16,597] INFO jute.maxbuffer value is 1048575 Bytes (org.apache.zookeeper.ClientCnxnSocket)
[2024-09-05 06:20:16,639] INFO zookeeper.request.timeout value is 0. feature enabled=false (org.apache.zookeeper.ClientCnxn)
[2024-09-05 06:20:16,713] INFO Opening socket connection to server zk-svc/10.151.0.57:2182. (org.apache.zookeeper.ClientCnxn)
[2024-09-05 06:20:16,745] INFO Socket connection established, initiating session, client: /10.151.6.21:39742, server: zk-svc/10.151.0.57:2182 (org.apache.zookeeper.ClientCnxn)
[2024-09-05 06:20:16,765] INFO Session establishment complete on server zk-svc/10.151.0.57:2182, session id = 0x2000003a6d6003e, negotiated timeout = 40000 (org.apache.zookeeper.ClientCnxn)
[2024-09-05 06:20:56,676] ERROR Timed out waiting for connection to Zookeeper server [zk-svc:2182]. (io.confluent.admin.utils.ClusterStatus)
[2024-09-05 06:20:56,792] INFO EventThread shut down for session: 0x2000003a6d6003e (org.apache.zookeeper.ClientCnxn)
[2024-09-05 06:20:56,793] INFO Session: 0x2000003a6d6003e closed (org.apache.zookeeper.ZooKeeper)
Using log4j config /etc/kafka/log4j.properties