Brokers restarting automatically

Hello All,
We have 3 brokers and 3 zookeepers pods configuration running in kubernetes. When we create topics (~300) brokers going down automatically and restarting continuously. In logs we found this error after broker restart.

INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /, server: kafka-cp-zookeeper-headless
[main] ERROR io.confluent.admin.utils.ClusterStatus - Timed out waiting for connection to Zookeeper server [kafka-cp-zookeeper-headless:2181].

[main-SendThread(kafka-cp-zookeeper-headless:2181)] WARN org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 40000ms for sessionid 0x0

[main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x0 closed

Is there any way we can get the cause for broker restart or any pointers for debugging issues?

Hey @naveen8384

welcome to the forum :slight_smile:

did you check the zookeeper pods?
according to your logs the issue seem to be related to zookeeper timeouts


1 Like

Hi @mmuehlbeyer ,
I checked zookeeper logs no errors and pods are running fine. I don’t see any restarts for zookeeper pods.

ok strange
any errors in the logs before the broker go down?

We found pod restarts happened due to OOMKilled error on Prometheus container. We increased memory and CPU for that and from then we are not seeing any restarts.