Hey,
let’s assume that I have a 3-node Kafka cluster and a 5-node ZooKeeper cluster. I wonder what would happen if the zookeeper quorum is lost (e.g. 3 zookeeper nodes are down). Since KIP-500 is in the room: I am referring to Kafka v2.7 or earlier.
My assumption is that the cluster remains writable and consumable for all producers and consumers. Additionally I assume that I won’t be able to create new topics, acls, quotas etc. I’m not sure what would happen if a broker dies now, as ZooKeeper is required for leader elections as far as I know.
Couldn’t find any documentation that goes into detail here.
@weeco The real impact to Kafka after a Zk cluster loses quorum is when there is an ISR change or update. So when there is a change in the state of the cluster, say a broker determines that a partition goes out of sync for any reason, then that broker needs to talk to Zookeeper to update the status of that partition, then the controller is notified, which then tell the other brokers to get the latest data from Zookeeper. If Zookeeper is down while all of this happens, broker ISR list becomes inaccurate.
In theory, as long as no changes occur on the brokers and as long as all the brokers are alive, clients will have NO impact while administrators work on bringing up the Zk quorum.