Brokers restarting automatically

Ops
1

Hello All,
We have 3 brokers and 3 zookeepers pods configuration running in kubernetes. When we create topics (~300) brokers going down automatically and restarting continuously. In logs we found this error after broker restart.

INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /, server: kafka-cp-zookeeper-headless
[main] ERROR io.confluent.admin.utils.ClusterStatus - Timed out waiting for connection to Zookeeper server [kafka-cp-zookeeper-headless:2181].

[main-SendThread(kafka-cp-zookeeper-headless:2181)] WARN org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 40000ms for sessionid 0x0

[main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x0 closed

Is there any way we can get the cause for broker restart or any pointers for debugging issues?

2

Hey @naveen8384

welcome to the forum :slight_smile:

did you check the zookeeper pods?
according to your logs the issue seem to be related to zookeeper timeouts

best,
michael

1 Like
3

Hi @mmuehlbeyer ,
I checked zookeeper logs no errors and pods are running fine. I don’t see any restarts for zookeeper pods.

4

ok strange
any errors in the logs before the broker go down?

5

We found pod restarts happened due to OOMKilled error on Prometheus container. We increased memory and CPU for that and from then we are not seeing any restarts.