# Brokers failing to connect to zookeeper **URL:** \ **Category:** Ops\ **Created:** [17 May 2022 17:12 UTC](https://forum.confluent.io/t/brokers-failing-to-connect-to-zookeeper/5086 "2022-05-17T17:12:55Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![naveen8384](https://avatars.discourse-cdn.com/v4/letter/n/da6949/32.png) [@naveen8384](https://forum.confluent.io/u/naveen8384)\ **Post date:** [17 May 2022 17:12 UTC](https://forum.confluent.io/t/brokers-failing-to-connect-to-zookeeper/5086/1 "2022-05-17T17:12:56Z")
Hello All, We are facing one issue when we restart both zookeepers and brokers at same time. We have a 3 brokers pods and 3 zookeeper pods running on kubernetes environment and when there is any activity on kubernetes environment or when we deploy helm chart after making any changes that causes zookeeper and brokers to recreate then we got this issue. In logs we can see that broker is trying to connect using zookeepers headless service but timing out. Then again we have to manually restart brokers and zookeepers to connect again. This is causing downtime whenever we are deploying new release. Is there any option to deploy without manually restarting pods? ---
**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\ **Post date:** [18 May 2022 05:21 UTC](https://forum.confluent.io/t/brokers-failing-to-connect-to-zookeeper/5086/2 "2022-05-18T05:21:52Z")
Hi @naveen8384 could provide some details about you env? I assume you’re using a helm chart to start the stack? if yes could you provide some details about the helm chart? best, michael ---
**Author:** ![naveen8384](https://avatars.discourse-cdn.com/v4/letter/n/da6949/32.png) [@naveen8384](https://forum.confluent.io/u/naveen8384)\ **Post date:** [18 May 2022 08:19 UTC](https://forum.confluent.io/t/brokers-failing-to-connect-to-zookeeper/5086/3 "2022-05-18T08:19:05Z")
Hi @mmuehlbeyer , we are using confluent helm charts, which are using zookeeper image version 5.5.3 and kafka image version 5.5.3. > **[GitHub - confluentinc/cp-helm-charts: The Confluent Platform Helm charts...](https://github.com/confluentinc/cp-helm-charts)** > > The Confluent Platform Helm charts enable you to deploy Confluent Platform services on Kubernetes for development, test, and proof of concept environments. - GitHub - confluentinc/cp-helm-charts: T... this are the logs we got in broker pod ```auto + export KAFKA_BROKER_ID=0 + exec /etc/confluent/docker/run ===> User uid=0(root) gid=0(root) groups=0(root) ===> Configuring ... ===> Running preflight checks ... ===> Check if /var/lib/kafka/data is writable ... ===> Check if Zookeeper is healthy ... [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:53 GMT [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=kafka-cp-kafka-0.kafka-cp-kafka-headless.kafka.svc.cluster.local [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_222 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Azul Systems, Inc. [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/zulu-8-amd64/jre [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/etc/confluent/docker/docker-utils.jar [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler= [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=5.4.148-1.el7.elrepo.x86_64 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/ [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=150MB [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=2276MB [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=153MB [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=kafka-cp-zookeeper-headless:2181 sessionTimeout=40000 watcher=io.confluent.admi n.utils.ZookeeperConnectionWatcher@65b3120a [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled= [main-SendThread(kafka-cp-zookeeper-headless:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server kafka-cp-zookeeper-headless/ :2181. Will not attempt to authenticate using SASL (unknown error) [main-SendThread(kafka-cp-zookeeper-headless:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /:3978 8, server: kafka-cp-zookeeper-headless/:2181 [main] ERROR io.confluent.admin.utils.ClusterStatus - Timed out waiting for connection to Zookeeper server [kafka-cp-zookeeper-headless:2181]. [main-SendThread(kafka-cp-zookeeper-headless:2181)] WARN org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 40000ms for sessionid 0x0 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x0 closed [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x0 ``` ---
**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\ **Post date:** [18 May 2022 08:23 UTC](https://forum.confluent.io/t/brokers-failing-to-connect-to-zookeeper/5086/4 "2022-05-18T08:23:08Z")
hi @naveen8384 thanks for the information. regarding > [@naveen8384](#): > > when we deploy helm chart after making any changes that causes zookeeper and brokers to recreate could you provide an example what you’re doing? best, michael ---
**Author:** ![naveen8384](https://avatars.discourse-cdn.com/v4/letter/n/da6949/32.png) [@naveen8384](https://forum.confluent.io/u/naveen8384)\ **Post date:** [18 May 2022 08:35 UTC](https://forum.confluent.io/t/brokers-failing-to-connect-to-zookeeper/5086/5 "2022-05-18T08:35:19Z")
Any changes which causes broker and zookeeper pods to recreate, like changing resource requests or limits for broker and zookeeper pods. ![image](https://us1.discourse-cdn.com/flex019/uploads/confluentcommunity/original/2X/0/07c30eeedc5e0bce23e17c14fb480173dacc927c.png) ---
**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\ **Post date:** [18 May 2022 08:43 UTC](https://forum.confluent.io/t/brokers-failing-to-connect-to-zookeeper/5086/6 "2022-05-18T08:43:33Z")
ok I see are you changing config like this? `helm upgrade cp-helm-charts` best, michael ---
**Author:** ![naveen8384](https://avatars.discourse-cdn.com/v4/letter/n/da6949/32.png) [@naveen8384](https://forum.confluent.io/u/naveen8384)\ **Post date:** [18 May 2022 08:48 UTC](https://forum.confluent.io/t/brokers-failing-to-connect-to-zookeeper/5086/7 "2022-05-18T08:48:10Z")
Yes, this is how we are deploying changes. ---
**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\ **Post date:** [18 May 2022 08:57 UTC](https://forum.confluent.io/t/brokers-failing-to-connect-to-zookeeper/5086/8 "2022-05-18T08:57:33Z")
ok did you check if the uri/servicename of the headless service changes? ---
**Author:** ![naveen8384](https://avatars.discourse-cdn.com/v4/letter/n/da6949/32.png) [@naveen8384](https://forum.confluent.io/u/naveen8384)\ **Post date:** [18 May 2022 10:10 UTC](https://forum.confluent.io/t/brokers-failing-to-connect-to-zookeeper/5086/9 "2022-05-18T10:10:33Z")
No, headless service name is same. only the existing pods deleted and new pods created. If we scale down brokers, zookeepers and scale up again then everything is working. ---
**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\ **Post date:** [18 May 2022 12:19 UTC](https://forum.confluent.io/t/brokers-failing-to-connect-to-zookeeper/5086/10 "2022-05-18T12:19:22Z")
strange there is a similar issue listed here > > > I'm trying to install Confluent Operator in our K8s cluster with Istio in it. Al…though the instruction wasn't included in their quick start guidelines, I'm hoping someone maybe came across with this problem. > > Here's the steps I've done: > 1. Install the cluster with minimum requirements of 10 nodes > 2. Install \`istio\` components > 2. Install the \`operator\` charts > 3. Install \`zookeeper\` charts > 4. Install \`kafka\` charts \<--- \*\*Always failed\*\* > > Errors based on kafka container: > \`\`\` > \[main-SendThread(zookeeper.operator.svc.cluster.local:2181)\] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper.operator.svc.cluster.local/46.19.XXX.XXX:2181. Will not attempt to authenticate using SASL (unknown error) > \[main-SendThread(zookeeper.operator.svc.cluster.local:2181)\] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /192.168.XX.XX:45356, server: zookeeper.operator.svc.cluster.local/46.19.XXX.XXX:2181 > \[main-SendThread(zookeeper.operator.svc.cluster.local:2181)\] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper.operator.svc.cluster.local/46.19.XXX.XXX:2181, unexpected error, closing socket connection and attempting reconnect > java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > at sun.nio.ch.IOUtil.read(IOUtil.java:192) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) > at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75) > at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) > > \[main\] ERROR io.confluent.admin.utils.ClusterStatus - Timed out waiting for connection to Zookeeper server \[zookeeper.operator.svc.cluster.local:2181/kafka-operator\]. > \[main-SendThread(zookeeper.operator.svc.cluster.local:2181)\] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper.operator.svc.cluster.local/46.19.XXX.XXX:2181. Will not attempt to authenticate using SASL (unknown error) > \[main-SendThread(zookeeper.operator.svc.cluster.local:2181)\] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /192.168.XX.XX:41440, server: zookeeper.operator.svc.cluster.local/46.19.XXX.XXX:2181 > \[main\] INFO org.apache.zookeeper.ZooKeeper - Session: 0x0 closed > \[main-EventThread\] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x0 > \`\`\` > > Tried some workaround with no luck. > 1. Install VirtualService to explicitly point to the \`zookeeper\` service > \`\`\` > cat \<\ apiVersion: networking.istio.io/v1alpha3 > kind: VirtualService > metadata: > name: zookeeper > spec: > hosts: > - zookeeper.operator.svc.cluster.local > http: > - name: zookeeper > route: > - destination: > host: zookeeper > \--- > EOF > \`\`\` > > 2. And also ServiceEntry for both \`zookeeper\` and \`kafka\` > \`\`\` > cat \<\ apiVersion: networking.istio.io/v1alpha3 > kind: ServiceEntry > metadata: > name: zookeeper > spec: > location: MESH\_INTERNAL > hosts: > - zookeeper.operator.svc.cluster.local > trafficPolicy: > tls: > mode: ISTIO\_MUTUAL > \--- > EOF > > > cat \<\ apiVersion: networking.istio.io/v1alpha3 > kind: ServiceEntry > metadata: > name: kafka > spec: > location: MESH\_INTERNAL > hosts: > - kafka.operator.svc.cluster.local > trafficPolicy: > tls: > mode: ISTIO\_MUTUAL > \--- > EOF > \`\`\` > > Is it related to any configuration of Confluent helm charts? > > Thanks! > > Reference page: \[Confluent Operator Quick Start\](https://docs.confluent.io/current/installation/operator/co-quickstart.html). according to the docs confluent 5.5.3 ships with zookeeper 3.5.7 and the issue talks about zookeeper version 3.5.8 so you might have hit the bug might be worth trying to test with confluent 6.x btw: any special reasions for using these version? best, michael ---
**Author:** ![naveen8384](https://avatars.discourse-cdn.com/v4/letter/n/da6949/32.png) [@naveen8384](https://forum.confluent.io/u/naveen8384)\ **Post date:** [19 May 2022 07:44 UTC](https://forum.confluent.io/t/brokers-failing-to-connect-to-zookeeper/5086/11 "2022-05-19T07:44:48Z")
Hi @mmuehlbeyer , Thanks for sharing the issue reference. I think this is the same issue which we are facing. There is no specific reason for using version 5.5.3. we have started with that version and still using the same version. I have few questions regarding upgrade. Can we update only the zookeeper version to 6.1 version from 3.5.8 keeping brokers version same? Is version 6.1 compatible with 3.5.8 data, we don’t want to lose existing data? ---
**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\ **Post date:** [19 May 2022 09:29 UTC](https://forum.confluent.io/t/brokers-failing-to-connect-to-zookeeper/5086/12 "2022-05-19T09:29:17Z")
hi @naveen8384 according to the docs confluent 7.1 also ships with zookeeper 3.5.x so I assume it should be working, but never tested by myself. [https://docs.confluent.io/platform/7.1.1/installation/upgrade.html#upgrade-zk](https://docs.confluent.io/platform/7.1.1/installation/upgrade.html#upgrade-zk) best, michael ---
**Author:** ![naveen8384](https://avatars.discourse-cdn.com/v4/letter/n/da6949/32.png) [@naveen8384](https://forum.confluent.io/u/naveen8384)\ **Post date:** [23 May 2022 06:44 UTC](https://forum.confluent.io/t/brokers-failing-to-connect-to-zookeeper/5086/13 "2022-05-23T06:44:10Z")
Hi @mmuehlbeyer , I tried to upgrade zookeeper version to 6.2.2 but we are not able to bring zookeepers up. In logs we got these errors. ```auto ===> User uid=1000(appuser) gid=1000(appuser) groups=1000(appuser) ===> Configuring ... [Errno 13] Permission denied: '/var/lib/zookeeper/data/myid' Command [/usr/local/bin/dub template /etc/confluent/docker/myid.template /var/lib/zookeeper/data/myid] FAILED ! 2022-03-16 08:55:12,639] ERROR Disk error while locking directory /opt/kafka/data-0/logs (kafka.server.LogDirFailureChannel) java.nio.file.AccessDeniedException: /opt/kafka/data-0/logs/.lock [2022-03-16 08:55:14,971] INFO Loading logs from log dirs ArraySeq(/opt/kafka/data-0/logs) (kafka.log.LogManager) [2022-03-16 08:55:14,975] INFO Skipping recovery for all logs in /opt/kafka/data-0/logs since clean shutdown file was found (kafka.log.LogManager) [2022-03-16 08:55:14,979] ERROR Error while loading log dir /opt/kafka/data-0/logs (kafka.log.LogManager) java.nio.file.AccessDeniedException: /opt/kafka/data-0/logs/.kafka_cleanshutdown at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) [2022-03-16 08:55:14,981] ERROR Error while loading log dir /opt/kafka/data-0/logs (kafka.server.LogDirFailureChannel) java.nio.file.AccessDeniedException: /opt/kafka/data-0/logs/.kafka_cleanshutdown ``` While analyzinglogs we found, old image 5.5.3 is using **root** user while new image is using **appuser** user. so we are not able to get old data. ```auto ===> User uid=0(root) gid=0(root) groups=0(root) ===> Configuring ... ===> Running preflight checks ... ===> Check if /var/lib/zookeeper/data is writable ... ===> Check if /var/lib/zookeeper/log is writable ... ===> Launching ... ===> Printing /var/lib/zookeeper/data/myid 1===> Launching zookeeper ... ``` can you please suggest a way to use existing data with new image? Thanks Naveen ---
**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\ **Post date:** [23 May 2022 06:51 UTC](https://forum.confluent.io/t/brokers-failing-to-connect-to-zookeeper/5086/14 "2022-05-23T06:51:02Z")
Hi @naveen8384 did you try to change security context in your values.yml ```auto securityContext: {} # runAsUser: 1000 # runAsGroup: 1000 ``` best, michael ---
**Author:** ![naveen8384](https://avatars.discourse-cdn.com/v4/letter/n/da6949/32.png) [@naveen8384](https://forum.confluent.io/u/naveen8384)\ **Post date:** [11 July 2022 08:11 UTC](https://forum.confluent.io/t/brokers-failing-to-connect-to-zookeeper/5086/15 "2022-07-11T08:11:39Z")
Hi @mmuehlbeyer , We updated the security context and successfully updated confluent version after which the zookeeper issue is resolved. Now we are able to apply rolling updates without any downtime. Once again Thank you for you support throughout the process.