KAFKA cluster with kraft

Hi, team

Environment: 2 vm server with confluent-community-7.2.2

###############################################################
=> Server 1 :
process.roles=broker,controller
ip: 10.20.10.4
=> Server 2 :
process.roles=broker,controller
ip: 10.20.10.5
###############################################################

I’m having difficulties running the KAFKA community with KRAFT. With only 1 controller quorum voters, it works normally. But when I add a new controller, it doesn’t work.

=> config work:
controller.quorum.voters=1@10.20.10.4:9093

=> config not work: controller.quorum.voters=1@10.20.10.4:9093,controller.quorum.voters=2@10.20.10.5:9093

Log not working:

[2022-10-31 20:16:24,812] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:24,936] INFO [BrokerLifecycleManager id=1] Unable to register the broker because the RPC got timed out before it could be sent. (kafka.server.BrokerLifecycleManager)
[2022-10-31 20:16:25,356] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:25,356] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:26,554] INFO [RaftManager nodeId=1] Re-elect as candidate after election backoff has completed (org.apache.kafka.raft.KafkaRaftClient)
[2022-10-31 20:16:26,558] INFO [RaftManager nodeId=1] Completed transition to CandidateState(localId=1, epoch=26, retries=26, electionTimeoutMs=1702) (org.apache.kafka.raft.QuorumState)
[2022-10-31 20:16:26,561] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:26,561] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:27,098] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:27,098] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:27,518] ERROR [BrokerLifecycleManager id=1] Shutting down because we were unable to register with the controller quorum. (kafka.server.BrokerLifecycleManager)
[2022-10-31 20:16:27,519] INFO [BrokerLifecycleManager id=1] registrationTimeout: shutting down event queue. (org.apache.kafka.queue.KafkaEventQueue)
[2022-10-31 20:16:27,520] INFO [BrokerLifecycleManager id=1] Transitioning from STARTING to SHUTTING_DOWN. (kafka.server.BrokerLifecycleManager)
[2022-10-31 20:16:27,521] INFO [BrokerServer id=1] Transition from STARTING to STARTED (kafka.server.BrokerServer)
[2022-10-31 20:16:27,521] INFO [BrokerToControllerChannelManager broker=1 name=heartbeat]: Shutting down (kafka.server.BrokerToControllerRequestThread)
[2022-10-31 20:16:27,523] INFO [BrokerToControllerChannelManager broker=1 name=heartbeat]: Shutdown completed (kafka.server.BrokerToControllerRequestThread)
[2022-10-31 20:16:27,526] ERROR [BrokerServer id=1] Fatal error during broker startup. Prepare to shutdown (kafka.server.BrokerServer)
java.util.concurrent.CancellationException
at java.base/java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2396)
at kafka.server.BrokerLifecycleManager$ShutdownEvent.run(BrokerLifecycleManager.scala:478)
at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:174)
at java.base/java.lang.Thread.run(Thread.java:829)
[2022-10-31 20:16:27,538] INFO [BrokerServer id=1] Transition from STARTED to SHUTTING_DOWN (kafka.server.BrokerServer)
[2022-10-31 20:16:27,539] INFO [BrokerServer id=1] shutting down (kafka.server.BrokerServer)
[2022-10-31 20:16:27,535] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:27,541] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:27,532] INFO Broker to controller channel manager for heartbeat shutdown (kafka.server.BrokerToControllerChannelManagerImpl)
[2022-10-31 20:16:27,528] INFO [BrokerToControllerChannelManager broker=1 name=heartbeat]: Stopped (kafka.server.BrokerToControllerRequestThread)
[2022-10-31 20:16:27,539] INFO [BrokerMetadataListener id=1] beginShutdown: shutting down event queue. (org.apache.kafka.queue.KafkaEventQueue)
[2022-10-31 20:16:27,545] INFO [SocketServer listenerType=BROKER, nodeId=1] Stopping socket server request processors (kafka.network.SocketServer)
[2022-10-31 20:16:27,551] INFO [SocketServer listenerType=BROKER, nodeId=1] Stopped socket server request processors (kafka.network.SocketServer)
[2022-10-31 20:16:27,552] INFO [data-plane Kafka Request Handler on Broker 1], shutting down (kafka.server.KafkaRequestHandlerPool)
[2022-10-31 20:16:27,560] INFO [data-plane Kafka Request Handler on Broker 1], shut down completely (kafka.server.KafkaRequestHandlerPool)
[2022-10-31 20:16:27,561] INFO [ExpirationReaper-1-AlterAcls]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,565] INFO [ExpirationReaper-1-AlterAcls]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,566] INFO [ExpirationReaper-1-AlterAcls]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,568] INFO [KafkaApi-1] Shutdown complete. (kafka.server.KafkaApis)
[2022-10-31 20:16:27,570] INFO [BrokerMetadataListener id=1] closed event queue. (org.apache.kafka.queue.KafkaEventQueue)
[2022-10-31 20:16:27,573] INFO [TransactionCoordinator id=1] Shutting down. (kafka.coordinator.transaction.TransactionCoordinator)
[2022-10-31 20:16:27,574] INFO [Transaction State Manager 1]: Shutdown complete (kafka.coordinator.transaction.TransactionStateManager)
[2022-10-31 20:16:27,574] INFO [Transaction Marker Channel Manager 1]: Shutting down (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2022-10-31 20:16:27,574] INFO [Transaction Marker Channel Manager 1]: Shutdown completed (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2022-10-31 20:16:27,576] INFO [TransactionCoordinator id=1] Shutdown complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2022-10-31 20:16:27,577] INFO [GroupCoordinator 1]: Shutting down. (kafka.coordinator.group.GroupCoordinator)
[2022-10-31 20:16:27,601] INFO [ExpirationReaper-1-Heartbeat]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,604] INFO [ExpirationReaper-1-Heartbeat]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,604] INFO [ExpirationReaper-1-Heartbeat]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,605] INFO [ExpirationReaper-1-Rebalance]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,606] INFO [ExpirationReaper-1-Rebalance]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,607] INFO [ExpirationReaper-1-Rebalance]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,608] INFO [GroupCoordinator 1]: Shutdown complete. (kafka.coordinator.group.GroupCoordinator)
[2022-10-31 20:16:27,609] INFO [ReplicaManager broker=1] Shutting down (kafka.server.ReplicaManager)
[2022-10-31 20:16:27,611] INFO [ReplicaFetcherManager on broker 1] shutting down (kafka.server.ReplicaFetcherManager)
[2022-10-31 20:16:27,614] INFO [ReplicaFetcherManager on broker 1] shutdown completed (kafka.server.ReplicaFetcherManager)
[2022-10-31 20:16:27,614] INFO [ReplicaAlterLogDirsManager on broker 1] shutting down (kafka.server.ReplicaAlterLogDirsManager)
[2022-10-31 20:16:27,615] INFO [ReplicaAlterLogDirsManager on broker 1] shutdown completed (kafka.server.ReplicaAlterLogDirsManager)
[2022-10-31 20:16:27,615] INFO [ExpirationReaper-1-Fetch]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,618] INFO [ExpirationReaper-1-Fetch]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,619] INFO [ExpirationReaper-1-Fetch]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,619] INFO [ExpirationReaper-1-Produce]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,632] INFO [ExpirationReaper-1-Produce]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,632] INFO [ExpirationReaper-1-Produce]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,633] INFO [ExpirationReaper-1-DeleteRecords]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,635] INFO [ExpirationReaper-1-DeleteRecords]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,635] INFO [ExpirationReaper-1-DeleteRecords]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,636] INFO [ExpirationReaper-1-ElectLeader]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,642] INFO [ExpirationReaper-1-ElectLeader]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,642] INFO [ExpirationReaper-1-ElectLeader]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,658] INFO [ReplicaManager broker=1] Shut down completely (kafka.server.ReplicaManager)
[2022-10-31 20:16:27,659] INFO [BrokerToControllerChannelManager broker=1 name=alterIsr]: Shutting down (kafka.server.BrokerToControllerRequestThread)
[2022-10-31 20:16:27,659] INFO [BrokerToControllerChannelManager broker=1 name=alterIsr]: Stopped (kafka.server.BrokerToControllerRequestThread)
[2022-10-31 20:16:27,659] INFO [BrokerToControllerChannelManager broker=1 name=alterIsr]: Shutdown completed (kafka.server.BrokerToControllerRequestThread)
[2022-10-31 20:16:27,664] INFO Broker to controller channel manager for alterIsr shutdown (kafka.server.BrokerToControllerChannelManagerImpl)
[2022-10-31 20:16:27,665] INFO [BrokerToControllerChannelManager broker=1 name=forwarding]: Shutting down (kafka.server.BrokerToControllerRequestThread)
[2022-10-31 20:16:27,666] INFO [BrokerToControllerChannelManager broker=1 name=forwarding]: Stopped (kafka.server.BrokerToControllerRequestThread)
[2022-10-31 20:16:27,668] INFO [BrokerToControllerChannelManager broker=1 name=forwarding]: Shutdown completed (kafka.server.BrokerToControllerRequestThread)
[2022-10-31 20:16:27,669] INFO Broker to controller channel manager for forwarding shutdown (kafka.server.BrokerToControllerChannelManagerImpl)
[2022-10-31 20:16:27,670] INFO Shutting down. (kafka.log.LogManager)
[2022-10-31 20:16:27,712] WARN /tmp/kraft-combined-logs/.kafka_cleanshutdown (kafka.utils.CoreUtils$)
java.nio.file.FileAlreadyExistsException: /tmp/kraft-combined-logs/.kafka_cleanshutdown
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:94)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
at java.base/java.nio.file.Files.newByteChannel(Files.java:371)
at java.base/java.nio.file.Files.createFile(Files.java:648)
at kafka.log.LogManager.$anonfun$shutdown$16(LogManager.scala:559)
at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:68)
at kafka.log.LogManager.$anonfun$shutdown$10(LogManager.scala:559)
at kafka.log.LogManager.$anonfun$shutdown$10$adapted(LogManager.scala:545)
at kafka.utils.Implicits$MapExtensionMethods$.$anonfun$forKeyValue$1(Implicits.scala:62)
at scala.collection.mutable.HashMap$Node.foreachEntry(HashMap.scala:633)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:499)
at kafka.log.LogManager.shutdown(LogManager.scala:545)
at kafka.server.BrokerServer.$anonfun$shutdown$21(BrokerServer.scala:519)
at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:68)
at kafka.server.BrokerServer.shutdown(BrokerServer.scala:519)
at kafka.server.BrokerServer.startup(BrokerServer.scala:443)
at kafka.server.KafkaRaftServer.$anonfun$startup$2(KafkaRaftServer.scala:114)
at kafka.server.KafkaRaftServer.$anonfun$startup$2$adapted(KafkaRaftServer.scala:114)
at scala.Option.foreach(Option.scala:437)
at kafka.server.KafkaRaftServer.startup(KafkaRaftServer.scala:114)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
[2022-10-31 20:16:27,716] INFO Shutdown complete. (kafka.log.LogManager)
[2022-10-31 20:16:27,717] INFO [ThrottledChannelReaper-Fetch]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,721] INFO [ThrottledChannelReaper-Fetch]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,722] INFO [ThrottledChannelReaper-Fetch]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,722] INFO [ThrottledChannelReaper-Produce]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,723] INFO [ThrottledChannelReaper-Produce]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,724] INFO [ThrottledChannelReaper-Produce]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,724] INFO [ThrottledChannelReaper-Request]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,725] INFO [ThrottledChannelReaper-Request]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,729] INFO [ThrottledChannelReaper-ControllerMutation]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,729] INFO [ThrottledChannelReaper-ControllerMutation]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,728] INFO [ThrottledChannelReaper-Request]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,730] INFO [ThrottledChannelReaper-ControllerMutation]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,733] INFO [SocketServer listenerType=BROKER, nodeId=1] Shutting down socket server (kafka.network.SocketServer)
[2022-10-31 20:16:27,808] INFO [SocketServer listenerType=BROKER, nodeId=1] Shutdown completed (kafka.network.SocketServer)
[2022-10-31 20:16:27,809] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics)
[2022-10-31 20:16:27,809] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics)
[2022-10-31 20:16:27,810] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics)
[2022-10-31 20:16:27,813] INFO Broker and topic stats closed (kafka.server.BrokerTopicStats)
[2022-10-31 20:16:27,814] INFO [BrokerLifecycleManager id=1] closed event queue. (org.apache.kafka.queue.KafkaEventQueue)
[2022-10-31 20:16:27,859] INFO App info kafka.server for 1 unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2022-10-31 20:16:27,860] INFO [BrokerServer id=1] shut down completed (kafka.server.BrokerServer)
[2022-10-31 20:16:27,860] INFO [BrokerServer id=1] Transition from SHUTTING_DOWN to SHUTDOWN (kafka.server.BrokerServer)
[2022-10-31 20:16:27,860] ERROR Exiting Kafka. (kafka.Kafka$)
[2022-10-31 20:16:27,862] INFO [kafka-raft-io-thread]: Shutting down (kafka.raft.KafkaRaftManager$RaftIoThread)
[2022-10-31 20:16:27,863] INFO [RaftManager nodeId=1] Beginning graceful shutdown (org.apache.kafka.raft.KafkaRaftClient)
[2022-10-31 20:16:28,111] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:28,111] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:28,711] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:28,711] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:29,191] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:29,191] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:29,626] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:29,626] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:30,134] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:30,134] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:30,691] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:30,692] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:31,298] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:31,298] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:31,917] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:31,917] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:32,426] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:32,426] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:32,867] WARN [RaftManager nodeId=1] Graceful shutdown timed out after 5000ms (org.apache.kafka.raft.KafkaRaftClient)
[2022-10-31 20:16:32,868] ERROR [kafka-raft-io-thread]: Graceful shutdown of RaftClient failed (kafka.raft.KafkaRaftManager$RaftIoThread)
java.util.concurrent.TimeoutException: Timeout expired before graceful shutdown completed
at org.apache.kafka.raft.KafkaRaftClient$GracefulShutdown.failWithTimeout(KafkaRaftClient.java:2411)
at org.apache.kafka.raft.KafkaRaftClient.maybeCompleteShutdown(KafkaRaftClient.java:2166)
at org.apache.kafka.raft.KafkaRaftClient.poll(KafkaRaftClient.java:2233)
at kafka.raft.KafkaRaftManager$RaftIoThread.doWork(RaftManager.scala:53)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
[2022-10-31 20:16:32,869] INFO [kafka-raft-io-thread]: Stopped (kafka.raft.KafkaRaftManager$RaftIoThread)
[2022-10-31 20:16:32,869] INFO [kafka-raft-io-thread]: Shutdown completed (kafka.raft.KafkaRaftManager$RaftIoThread)
[2022-10-31 20:16:32,891] INFO [kafka-raft-outbound-request-thread]: Shutting down (kafka.raft.RaftSendThread)
[2022-10-31 20:16:32,891] INFO [kafka-raft-outbound-request-thread]: Stopped (kafka.raft.RaftSendThread)
[2022-10-31 20:16:32,892] INFO [kafka-raft-outbound-request-thread]: Shutdown completed (kafka.raft.RaftSendThread)
[2022-10-31 20:16:32,903] INFO [ControllerServer id=1] shutting down (kafka.server.ControllerServer)
[2022-10-31 20:16:32,904] INFO [SocketServer listenerType=CONTROLLER, nodeId=1] Stopping socket server request processors (kafka.network.SocketServer)
[2022-10-31 20:16:32,926] INFO [SocketServer listenerType=CONTROLLER, nodeId=1] Stopped socket server request processors (kafka.network.SocketServer)
[2022-10-31 20:16:32,927] INFO [Controller 1] QuorumController#beginShutdown: shutting down event queue. (org.apache.kafka.queue.KafkaEventQueue)
[2022-10-31 20:16:32,927] INFO [SocketServer listenerType=CONTROLLER, nodeId=1] Shutting down socket server (kafka.network.SocketServer)
[2022-10-31 20:16:32,934] INFO [SocketServer listenerType=CONTROLLER, nodeId=1] Shutdown completed (kafka.network.SocketServer)
[2022-10-31 20:16:32,935] INFO [data-plane Kafka Request Handler on Broker 1], shutting down (kafka.server.KafkaRequestHandlerPool)
[2022-10-31 20:16:32,937] INFO [data-plane Kafka Request Handler on Broker 1], shut down completely (kafka.server.KafkaRequestHandlerPool)
[2022-10-31 20:16:32,938] INFO [ExpirationReaper-1-AlterAcls]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:32,938] INFO [ExpirationReaper-1-AlterAcls]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:32,939] INFO [ExpirationReaper-1-AlterAcls]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:32,940] INFO [ThrottledChannelReaper-Fetch]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,940] INFO [ThrottledChannelReaper-Fetch]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,940] INFO [ThrottledChannelReaper-Fetch]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,940] INFO [ThrottledChannelReaper-Produce]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,950] INFO [ThrottledChannelReaper-Produce]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,950] INFO [ThrottledChannelReaper-Produce]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,950] INFO [ThrottledChannelReaper-Request]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,951] INFO [ThrottledChannelReaper-Request]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,952] INFO [ThrottledChannelReaper-Request]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,952] INFO [ThrottledChannelReaper-ControllerMutation]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,953] INFO [ThrottledChannelReaper-ControllerMutation]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,953] INFO [ThrottledChannelReaper-ControllerMutation]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,953] INFO [Controller 1] closed event queue. (org.apache.kafka.queue.KafkaEventQueue)
[2022-10-31 20:16:32,956] INFO App info kafka.server for 1 unregistered (org.apache.kafka.common.utils.AppInfoParser)

I am facing exactly same issue with 3 controller quorum voters.

There is one catch, if I race through to start all the three controllers in a few seconds, it works.

But starting one controller at a time with a few minutes apart always complain as below:
"Shutting down because we were unable to register with the controller quorum. "

As I reach to start the second controller, the first controller is down by that time…

How did you set up the listeners? Is there a separate listener for the controller on port 9093? Have you configured the controller.listener.names to list the dedicated controller listener defined in the listeners config?

Hello ver-alex

Thanks for your help.

Please see my server.properties

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@10.20.10.4:9093,2@10.20.10.5:9093
listeners=PLAINTEXT://10.20.10.4:9092,CONTROLLER://10.20.10.4:9093
inter.broker.listener.name=PLAINTEXT
advertised.listeners=PLAINTEXT://10.20.10.4:9092
controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
log.dirs=/tmp/server1/kraft-combined-logs

Hi, We managed to solve!

On first access, all nodes must be configured and active. After the first access, it votes the quorum and they can take down the other nodes.