Hello Kafka Community,
I’m currently facing a set of challenges with my Kafka and ZooKeeper setup on an Azure Ubuntu VM. I would greatly appreciate your insights and guidance on the following issues:
1. ZooKeeper Cluster ID and DataDir: After each VM restart, ZooKeeper generates a new cluster ID, causing synchronization problems with Kafka. I understand that changing the dataDir to a persistent location can resolve this. I have updated it to /var/zookeeper , but I’m open to suggestions and best practices.
Zookeeper Configuration
Systemctl status Zookeeper:
**root@CNAS-STACKSTORM-DEV-1:/home/azureuser# systemctl status zookeeper**
**● zookeeper.service**
** Loaded: loaded (/etc/systemd/system/zookeeper.service; enabled; vendor preset: enabled)**
** Active: failed (Result: exit-code) since Mon 2023-11-20 06:55:51 UTC; 5s ago**
** Process: 99111 ExecStart=/home/kafka/kafka/bin/zookeeper-server-start.sh /home/kafka/kafka/config/zookeeper.properties (code=exited, status=3)**
** Main PID: 99111 (code=exited, status=3)**
**Nov 20 06:55:51 CNAS-STACKSTORM-DEV-1 zookeeper-server-start.sh[99111]: at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperSe>**
**Nov 20 06:55:51 CNAS-STACKSTORM-DEV-1 zookeeper-server-start.sh[99111]: at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeepe>**
**Nov 20 06:55:51 CNAS-STACKSTORM-DEV-1 zookeeper-server-start.sh[99111]: at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.>**
**Nov 20 06:55:51 CNAS-STACKSTORM-DEV-1 zookeeper-server-start.sh[99111]: at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(Quorum>**
**Nov 20 06:55:51 CNAS-STACKSTORM-DEV-1 zookeeper-server-start.sh[99111]: at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.jav>**
**Nov 20 06:55:51 CNAS-STACKSTORM-DEV-1 zookeeper-server-start.sh[99111]: Unable to access datadir, exiting abnormally**
**Nov 20 06:55:51 CNAS-STACKSTORM-DEV-1 zookeeper-server-start.sh[99111]: [2023-11-20 06:55:51,657] INFO ZooKeeper audit is disabled. (org.apache.zookeeper.au>**
**Nov 20 06:55:51 CNAS-STACKSTORM-DEV-1 zookeeper-server-start.sh[99111]: [2023-11-20 06:55:51,658] ERROR Exiting JVM with code 3 (org.apache.zookeeper.util.S>**
**Nov 20 06:55:51 CNAS-STACKSTORM-DEV-1 systemd[1]: zookeeper.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED**
**Nov 20 06:55:51 CNAS-STACKSTORM-DEV-1 systemd[1]: zookeeper.service: Failed with result 'exit-code'.**
2. Kafka Topic Deletion: I’ve observed that all of my Kafka topics are getting deleted. I want to ensure that topics persist across restarts. Is there a specific Kafka broker configuration I should review? The delete.topic.enable property in server.properties is currently set to false
3. Kafka Producer Error: When attempting to publish messages using the Kafka console producer, I encounter the following error:
ERROR Error when sending message to topic XXX with key: null, value: X bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
I’ve verified my producer and broker configurations, but the issue persists. Any insights or suggestions on troubleshooting this would be immensely helpful.
Thank you in advance for your assistance!