# While creating the cluster, we are encountering issues with the 3 controllers **URL:** \ **Category:** Ops\ **Created:** [6 April 2025 05:50 UTC](https://forum.confluent.io/t/while-creating-the-cluster-we-are-encountering-issues-with-the-3-controllers/37642 "2025-04-06T05:50:16Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![naveen](https://avatars.discourse-cdn.com/v4/letter/n/f07891/32.png) [@naveen](https://forum.confluent.io/u/naveen)\ **Post date:** [6 April 2025 05:50 UTC](https://forum.confluent.io/t/while-creating-the-cluster-we-are-encountering-issues-with-the-3-controllers/37642/1 "2025-04-06T05:50:16Z")
**Dear Team,** While creating the KRaft cluster with 3 controllers and 3 brokers using the Confluent Operator, we created the controller resource definition file as shown below. However, upon startup, we are continuously encountering the exceptions mentioned below. Could you please review and suggest what might be wrong? Controller file:- apiVersion: [platform.confluent.io/v1beta1](http://platform.confluent.io/v1beta1) kind: KRaftController metadata: name: kraftcontroller namespace: kafka spec: oneReplicaPerNode: false dataVolumeCapacity: 10G image: application: [docker.io/confluentinc/cp-server:7.9.0](http://docker.io/confluentinc/cp-server:7.9.0) init: confluentinc/confluent-init-container:2.11.0 replicas: 3 configOverrides: server: - broker.id=1 - node.id=1 - cluster.id=ZbD3VucSSRupm5dJlDOwYQ - confluent.balancer.enable=true - confluent.operator.managed=true - controller.listener.names=CONTROLLER - controller.quorum.voters=1@kraftcontroller-0.kraftcontroller.kafka.svc.cluster.local:9074,2@kraftcontroller-1.kraftcontroller.kafka.svc.cluster.local:9074,3@kraftcontroller-2.kraftcontroller.kafka.svc.cluster.local:9074 - inter.broker.listener.name=REPLICATION - listener.security.protocol.map=CONTROLLER:PLAINTEXT,REPLICATION:PLAINTEXT - listeners=CONTROLLER://:9074 - log.dirs=/tmp/datalogs - log.message.format.version=3.4 - num.network.threads=4 - process.roles=controller - advertised.listeners=CONTROLLER://kraftcontroller-0:9074 **Exception logs** ToCatchUp - [MetadataLoader id=1] initializeNewPublishers: the loader is still catching up because we still don’t know the high water mark yet. [INFO] 2025-04-06 05:50:36,913 [kafka-1-raft-io-thread] org.apache.kafka.raft.KafkaRaftClient handleVoteRequest - [RaftManager id=1] Candidate sent a voter key (Optional[ReplicaKey(id=2, directoryId=Optional.empty)]) in the VOTE request that doesn’t match the local key (OptionalInt[1], 5pVL1Rivio2nNjWN6andVw); rejecting the vote [INFO] 2025-04-06 05:50:36,933 [kafka-1-raft-io-thread] org.apache.kafka.raft.KafkaRaftClient handleVoteRequest - [RaftManager id=1] Candidate sent a voter key (Optional[ReplicaKey(id=2, directoryId=Optional.empty)]) in the VOTE request that doesn’t match the local key (OptionalInt[1], 5pVL1Rivio2nNjWN6andVw); rejecting the vote [INFO] 2025-04-06 05:50:36,957 [kafka-1-raft-io-thread] org.apache.kafka.raft.KafkaRaftClient handleVoteRequest - [RaftManager id=1] Candidate sent a voter key (Optional[ReplicaKey(id=2, directoryId=Optional.empty)]) in the VOTE request that doesn’t match the local key (OptionalInt[1], 5pVL1Rivio2nNjWN6andVw); rejecting the vote [INFO] 2025-04-06 05:50:36,975 [kafka-1-raft-io-thread] org.apache.kafka.raft.KafkaRaftClient handleVoteRequest - [RaftManager id=1] Candidate sent a voter key (Optional[ReplicaKey(id=2, directoryId=Optional.empty)]) in the VOTE request that doesn’t match the local key (OptionalInt[1], 5pVL1Rivio2nNjWN6andVw); rejecting the vote [INFO] 2025-04-06 05:50:36,980 [kafka-1-raft-io-thread] org.apache.kafka.raft.KafkaRaftClient handleVoteRequest - [RaftManager id=1] Candidate sent a voter key (Optional[ReplicaKey(id=2, directoryId=Optional.empty)]) in the VOTE request that doesn’t match the local key (OptionalInt[1], 5pVL1Rivio2nNjWN6andVw); rejecting the vote [INFO] 2025-04-06 05:50:36,996 [kafka-1-raft-io-thread] org.apache.kafka.raft.KafkaRaftClient handleVoteRequest - [RaftManager id=1] Candidate sent a voter key (Optional[ReplicaKey(id=2, directoryId=Optional.empty)]) in the VOTE request that doesn’t match the local key (OptionalInt[1], 5pVL1Rivio2nNjWN6andVw); rejecting the vote [INFO] 2025-04-06 05:50:36,998 [kafka-1-raft-io-thread] org.apache.kafka.raft.KafkaRaftClient handleVoteRequest - [RaftManager id=1] Candidate sent a voter key (Optional[ReplicaKey(id=2, directoryId=Optional.empty)]) in the VOTE request that doesn’t match the local key (OptionalInt[1], 5pVL1Rivio2nNjWN6andVw); rejecting the vote [INFO] 2025-04-06 05:50:37,007 [kafka-1-metadata-loader-event-handler] org.apache.kafka.image.loader.MetadataLoader stillNeedToCatchUp - [MetadataLoader id=1] initializeNewPublishers: the loader is still catching up because we still don’t know the high water mark yet. [INFO] 2025-04-06 05:50:37,016 [kafka-1-raft-io-thread] org.apache.kafka.raft.KafkaRaftClient handleVoteRequest - [RaftManager id=1] Candidate sent a voter key (Optional[ReplicaKey(id=2, directoryId=Optional.empty)]) in the VOTE request that doesn’t match the local key (OptionalInt[1], 5pVL1Rivio2nNjWN6andVw); rejecting the vote [INFO] 2025-04-06 05:50:37,041 [kafka-1-raft-io-thread] org.apache.kafka.raft.KafkaRaftClient handleVoteRequest - [RaftManager id=1] Candidate sent a voter key (Optional[ReplicaKey(id=2, directoryId=Optional.empty)]) in the VOTE request that doesn’t match the local key (OptionalInt[1], 5pVL1Rivio2nNjWN6andVw); rejecting the vote [INFO] 2025-04-06 05:51:17,768 [kafka-1-metadata-loader-event-handler] org.apache.kafka.image.loader.MetadataLoader stillNeedToCatchUp - [MetadataLoader id=1] initializeNewPublishers: the loader is still catching up because we still don’t know the high water mark yet. [INFO] 2025-04-06 05:51:17,799 [kafka-1-raft-io-thread] org.apache.kafka.raft.KafkaRaftClient handleVoteRequest - [RaftManager id=1] Candidate sent a voter key (Optional[ReplicaKey(id=3, directoryId=Optional.empty)]) in the VOTE request that doesn’t match the local key (OptionalInt[1], KmAWIt-XQzC4HhtAEvsNFA); rejecting the vote [INFO] 2025-04-06 05:51:17,805 [kafka-1-raft-io-thread] org.apache.kafka.raft.KafkaRaftClient handleVoteRequest - [RaftManager id=1] Candidate sent a voter key (Optional[ReplicaKey(id=3, directoryId=Optional.empty)]) in the VOTE request that doesn’t match the local key (OptionalInt[1], KmAWIt-XQzC4HhtAEvsNFA); rejecting the vote [INFO] 2025-04-06 05:51:17,819 [kafka-1-raft-io-thread] org.apache.kafka.raft.KafkaRaftClient handleVoteRequest - [RaftManager id=1] Candidate sent a voter key (Optional[ReplicaKey(id=3, directoryId=Optional.empty)]) in the VOTE request that doesn’t match the local key (OptionalInt[1], KmAWIt-XQzC4HhtAEvsNFA); rejecting the vote [INFO] 2025-04-06 05:51:17,825 [kafka-1-raft-io-thread] org.apache.kafka.raft.KafkaRaftClient handleVoteRequest - [RaftManager id=1] Candidate sent a voter key (Optional[ReplicaKey(id=3, directoryId=Optional.empty)]) in the VOTE request that doesn’t match the local key (OptionalInt[1], KmAWIt-XQzC4HhtAEvsNFA); rejecting the vote ---
**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\ **Post date:** [7 April 2025 09:27 UTC](https://forum.confluent.io/t/while-creating-the-cluster-we-are-encountering-issues-with-the-3-controllers/37642/2 "2025-04-07T09:27:52Z")
hey @naveen welcome to the forum 🙂 (moving the thread to #Ops ) question regarding your conf: there are a lot of overrides, some of them are not necessary. is there any special reason for using them? best, michael ---
**Author:** ![naveen](https://avatars.discourse-cdn.com/v4/letter/n/f07891/32.png) [@naveen](https://forum.confluent.io/u/naveen)\ **Post date:** [8 April 2025 00:25 UTC](https://forum.confluent.io/t/while-creating-the-cluster-we-are-encountering-issues-with-the-3-controllers/37642/3 "2025-04-08T00:25:55Z")
Hi @ [mmuehlbeyer] Thank you for the reply. There’s no need to retain all the properties we were experimenting with while setting up the 3-node KRaft controller cluster. However, the cluster is failing to elect a leader and form a quorum. Do you have any suggestions on how we can successfully form the quorum in this 3-node controller setup? ---
**Author:** ![naveen](https://avatars.discourse-cdn.com/v4/letter/n/f07891/32.png) [@naveen](https://forum.confluent.io/u/naveen)\ **Post date:** [8 April 2025 05:36 UTC](https://forum.confluent.io/t/while-creating-the-cluster-we-are-encountering-issues-with-the-3-controllers/37642/4 "2025-04-08T05:36:23Z")
**I tried creating three separate controller YAML files, but I’m facing a cluster inconsistency issue due to each pod having a different `cluster.id` in the `meta.properties` file after a restart. How can I ensure that all pods in the Kubernetes cluster use the same `cluster.id` in their `meta.properties` file?** [ERROR] 2025-04-08 05:26:57,503 [kafka-1-raft-io-thread] org.apache.kafka.raft.KafkaRaftClient handleUnexpectedError - [RaftManager id=1] Unexpected error INCONSISTENT\_CLUSTER\_ID in VOTE response: InboundResponse(correlationId=3980, data=VoteResponseData(errorCode=104, topics=, nodeEndpoints=), source=kraftcontroller-1-0.kraftcontroller-1.kafka.svc.cluster.local:9074 (id: 2 rack: null)) ---
**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\ **Post date:** [8 April 2025 06:32 UTC](https://forum.confluent.io/t/while-creating-the-cluster-we-are-encountering-issues-with-the-3-controllers/37642/5 "2025-04-08T06:32:14Z")
cfk should take care of this did you try with the following quickstart ? > **[confluent-kubernetes-examples/quickstart-deploy at master ·...](https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/quickstart-deploy)** > > Example scenario workflows for Confluent for Kubernetes - confluentinc/confluent-kubernetes-examples just to have a setup up & running? best, michael ---
**Author:** ![naveen](https://avatars.discourse-cdn.com/v4/letter/n/f07891/32.png) [@naveen](https://forum.confluent.io/u/naveen)\ **Post date:** [8 April 2025 07:30 UTC](https://forum.confluent.io/t/while-creating-the-cluster-we-are-encountering-issues-with-the-3-controllers/37642/6 "2025-04-08T07:30:25Z")
We have successfully tested a single-node setup with all components running. However, while setting up a multi-node cluster with 3 controllers and 3 brokers, we are encountering issues. ---
**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\ **Post date:** [8 April 2025 07:36 UTC](https://forum.confluent.io/t/while-creating-the-cluster-we-are-encountering-issues-with-the-3-controllers/37642/7 "2025-04-08T07:36:09Z")
I see so what about this example? > ---
**Author:** ![naveen](https://avatars.discourse-cdn.com/v4/letter/n/f07891/32.png) [@naveen](https://forum.confluent.io/u/naveen)\ **Post date:** [14 April 2025 11:05 UTC](https://forum.confluent.io/t/while-creating-the-cluster-we-are-encountering-issues-with-the-3-controllers/37642/8 "2025-04-14T11:05:01Z")
Thank you for your response. The cluster is now working as expected. However, I am facing an issue while mounting external storage. The controller was successfully bound to the external PVC, but the broker is unable to mount the volume. I’m getting the following exception — any suggestions? [ERROR] 2025-04-14 10:59:18,163 [kafka-0-metadata-loader-event-handler] org.apache.kafka.server.fault.ProcessTerminatingFaultHandler handleFault - Encountered fatal fault: Error starting LogManager org.apache.kafka.common.KafkaException: Found directory /mnt/data/data0, ‘data0’ is not in the form of topic-partition or topic-partition.uniqueId-delete (if marked for deletion). Kafka’s log directories (and children) should only contain Kafka topic data. [ERROR] 2025-04-14 09:22:37,252 [kafka-0-metadata-loader-event-handler] org.apache.kafka.server.fault.ProcessTerminatingFaultHandler handleFault - Encountered fatal fault: Error starting LogManager org.apache.kafka.common.KafkaException: Found directory /mnt/data/data0/logs/lost+found, ‘lost+found’ is not in the form of topic-partition or topic-partition.uniqueId-delete (if marked for deletion). Kafka’s log directories (and children) should only contain Kafka topic data ---
**Author:** ![mmuehlbeyer](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mmuehlbeyer/32/1088_2.png) [@mmuehlbeyer](https://forum.confluent.io/u/mmuehlbeyer)\ **Post date:** [14 April 2025 11:47 UTC](https://forum.confluent.io/t/while-creating-the-cluster-we-are-encountering-issues-with-the-3-controllers/37642/9 "2025-04-14T11:47:54Z")
obviously it’s complaining about the the directories in youre PVC respectively mounted disks how does your server.properties look like? how did you attach the PVC? could you share the respective conf? ---
**Author:** ![naveen](https://avatars.discourse-cdn.com/v4/letter/n/f07891/32.png) [@naveen](https://forum.confluent.io/u/naveen)\ **Post date:** [14 April 2025 12:34 UTC](https://forum.confluent.io/t/while-creating-the-cluster-we-are-encountering-issues-with-the-3-controllers/37642/10 "2025-04-14T12:34:53Z")
Thanks for your response. There was a mismatch between the volume mount paths and the log directories that Kafka was expecting. I’ve fixed the issue. Now, after restarting or deleting the pods, all the data persists on our volumes as expected. Once again, thank you very much! ---
**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/confluentcommunity/original/1X/c49438c90c9df282e9996fdf6971be890c71b65a.svg) [@system](https://forum.confluent.io/u/system)\ **Post date:** [10 June 2025 19:03 UTC](https://forum.confluent.io/t/while-creating-the-cluster-we-are-encountering-issues-with-the-3-controllers/37642/11 "2025-06-10T19:03:52Z")
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.