KAFKA cluster with kraft

fernandofariadev · 31 October 2022 20:18

Hi, team

Environment: 2 vm server with confluent-community-7.2.2

###############################################################
=> Server 1 :
process.roles=broker,controller
ip: 10.20.10.4
=> Server 2 :
process.roles=broker,controller
ip: 10.20.10.5
###############################################################

I’m having difficulties running the KAFKA community with KRAFT. With only 1 controller quorum voters, it works normally. But when I add a new controller, it doesn’t work.

=> config work:
controller.quorum.voters=1@10.20.10.4:9093

=> config not work: controller.quorum.voters=1@10.20.10.4:9093,controller.quorum.voters=2@10.20.10.5:9093

Log not working:

[2022-10-31 20:16:24,812] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:24,936] INFO [BrokerLifecycleManager id=1] Unable to register the broker because the RPC got timed out before it could be sent. (kafka.server.BrokerLifecycleManager)
[2022-10-31 20:16:25,356] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:25,356] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:26,554] INFO [RaftManager nodeId=1] Re-elect as candidate after election backoff has completed (org.apache.kafka.raft.KafkaRaftClient)
[2022-10-31 20:16:26,558] INFO [RaftManager nodeId=1] Completed transition to CandidateState(localId=1, epoch=26, retries=26, electionTimeoutMs=1702) (org.apache.kafka.raft.QuorumState)
[2022-10-31 20:16:26,561] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:26,561] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:27,098] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:27,098] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:27,518] ERROR [BrokerLifecycleManager id=1] Shutting down because we were unable to register with the controller quorum. (kafka.server.BrokerLifecycleManager)
[2022-10-31 20:16:27,519] INFO [BrokerLifecycleManager id=1] registrationTimeout: shutting down event queue. (org.apache.kafka.queue.KafkaEventQueue)
[2022-10-31 20:16:27,520] INFO [BrokerLifecycleManager id=1] Transitioning from STARTING to SHUTTING_DOWN. (kafka.server.BrokerLifecycleManager)
[2022-10-31 20:16:27,521] INFO [BrokerServer id=1] Transition from STARTING to STARTED (kafka.server.BrokerServer)
[2022-10-31 20:16:27,521] INFO [BrokerToControllerChannelManager broker=1 name=heartbeat]: Shutting down (kafka.server.BrokerToControllerRequestThread)
[2022-10-31 20:16:27,523] INFO [BrokerToControllerChannelManager broker=1 name=heartbeat]: Shutdown completed (kafka.server.BrokerToControllerRequestThread)
[2022-10-31 20:16:27,526] ERROR [BrokerServer id=1] Fatal error during broker startup. Prepare to shutdown (kafka.server.BrokerServer)
java.util.concurrent.CancellationException
at java.base/java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2396)
at kafka.server.BrokerLifecycleManager$ShutdownEvent.run(BrokerLifecycleManager.scala:478)
at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:174)
at java.base/java.lang.Thread.run(Thread.java:829)
[2022-10-31 20:16:27,538] INFO [BrokerServer id=1] Transition from STARTED to SHUTTING_DOWN (kafka.server.BrokerServer)
[2022-10-31 20:16:27,539] INFO [BrokerServer id=1] shutting down (kafka.server.BrokerServer)
[2022-10-31 20:16:27,535] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:27,541] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:27,532] INFO Broker to controller channel manager for heartbeat shutdown (kafka.server.BrokerToControllerChannelManagerImpl)
[2022-10-31 20:16:27,528] INFO [BrokerToControllerChannelManager broker=1 name=heartbeat]: Stopped (kafka.server.BrokerToControllerRequestThread)
[2022-10-31 20:16:27,539] INFO [BrokerMetadataListener id=1] beginShutdown: shutting down event queue. (org.apache.kafka.queue.KafkaEventQueue)
[2022-10-31 20:16:27,545] INFO [SocketServer listenerType=BROKER, nodeId=1] Stopping socket server request processors (kafka.network.SocketServer)
[2022-10-31 20:16:27,551] INFO [SocketServer listenerType=BROKER, nodeId=1] Stopped socket server request processors (kafka.network.SocketServer)
[2022-10-31 20:16:27,552] INFO [data-plane Kafka Request Handler on Broker 1], shutting down (kafka.server.KafkaRequestHandlerPool)
[2022-10-31 20:16:27,560] INFO [data-plane Kafka Request Handler on Broker 1], shut down completely (kafka.server.KafkaRequestHandlerPool)
[2022-10-31 20:16:27,561] INFO [ExpirationReaper-1-AlterAcls]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,565] INFO [ExpirationReaper-1-AlterAcls]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,566] INFO [ExpirationReaper-1-AlterAcls]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,568] INFO [KafkaApi-1] Shutdown complete. (kafka.server.KafkaApis)
[2022-10-31 20:16:27,570] INFO [BrokerMetadataListener id=1] closed event queue. (org.apache.kafka.queue.KafkaEventQueue)
[2022-10-31 20:16:27,573] INFO [TransactionCoordinator id=1] Shutting down. (kafka.coordinator.transaction.TransactionCoordinator)
[2022-10-31 20:16:27,574] INFO [Transaction State Manager 1]: Shutdown complete (kafka.coordinator.transaction.TransactionStateManager)
[2022-10-31 20:16:27,574] INFO [Transaction Marker Channel Manager 1]: Shutting down (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2022-10-31 20:16:27,574] INFO [Transaction Marker Channel Manager 1]: Shutdown completed (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2022-10-31 20:16:27,576] INFO [TransactionCoordinator id=1] Shutdown complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2022-10-31 20:16:27,577] INFO [GroupCoordinator 1]: Shutting down. (kafka.coordinator.group.GroupCoordinator)
[2022-10-31 20:16:27,601] INFO [ExpirationReaper-1-Heartbeat]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,604] INFO [ExpirationReaper-1-Heartbeat]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,604] INFO [ExpirationReaper-1-Heartbeat]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,605] INFO [ExpirationReaper-1-Rebalance]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,606] INFO [ExpirationReaper-1-Rebalance]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,607] INFO [ExpirationReaper-1-Rebalance]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,608] INFO [GroupCoordinator 1]: Shutdown complete. (kafka.coordinator.group.GroupCoordinator)
[2022-10-31 20:16:27,609] INFO [ReplicaManager broker=1] Shutting down (kafka.server.ReplicaManager)
[2022-10-31 20:16:27,611] INFO [ReplicaFetcherManager on broker 1] shutting down (kafka.server.ReplicaFetcherManager)
[2022-10-31 20:16:27,614] INFO [ReplicaFetcherManager on broker 1] shutdown completed (kafka.server.ReplicaFetcherManager)
[2022-10-31 20:16:27,614] INFO [ReplicaAlterLogDirsManager on broker 1] shutting down (kafka.server.ReplicaAlterLogDirsManager)
[2022-10-31 20:16:27,615] INFO [ReplicaAlterLogDirsManager on broker 1] shutdown completed (kafka.server.ReplicaAlterLogDirsManager)
[2022-10-31 20:16:27,615] INFO [ExpirationReaper-1-Fetch]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,618] INFO [ExpirationReaper-1-Fetch]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,619] INFO [ExpirationReaper-1-Fetch]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,619] INFO [ExpirationReaper-1-Produce]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,632] INFO [ExpirationReaper-1-Produce]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,632] INFO [ExpirationReaper-1-Produce]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,633] INFO [ExpirationReaper-1-DeleteRecords]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,635] INFO [ExpirationReaper-1-DeleteRecords]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,635] INFO [ExpirationReaper-1-DeleteRecords]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,636] INFO [ExpirationReaper-1-ElectLeader]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,642] INFO [ExpirationReaper-1-ElectLeader]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,642] INFO [ExpirationReaper-1-ElectLeader]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:27,658] INFO [ReplicaManager broker=1] Shut down completely (kafka.server.ReplicaManager)
[2022-10-31 20:16:27,659] INFO [BrokerToControllerChannelManager broker=1 name=alterIsr]: Shutting down (kafka.server.BrokerToControllerRequestThread)
[2022-10-31 20:16:27,659] INFO [BrokerToControllerChannelManager broker=1 name=alterIsr]: Stopped (kafka.server.BrokerToControllerRequestThread)
[2022-10-31 20:16:27,659] INFO [BrokerToControllerChannelManager broker=1 name=alterIsr]: Shutdown completed (kafka.server.BrokerToControllerRequestThread)
[2022-10-31 20:16:27,664] INFO Broker to controller channel manager for alterIsr shutdown (kafka.server.BrokerToControllerChannelManagerImpl)
[2022-10-31 20:16:27,665] INFO [BrokerToControllerChannelManager broker=1 name=forwarding]: Shutting down (kafka.server.BrokerToControllerRequestThread)
[2022-10-31 20:16:27,666] INFO [BrokerToControllerChannelManager broker=1 name=forwarding]: Stopped (kafka.server.BrokerToControllerRequestThread)
[2022-10-31 20:16:27,668] INFO [BrokerToControllerChannelManager broker=1 name=forwarding]: Shutdown completed (kafka.server.BrokerToControllerRequestThread)
[2022-10-31 20:16:27,669] INFO Broker to controller channel manager for forwarding shutdown (kafka.server.BrokerToControllerChannelManagerImpl)
[2022-10-31 20:16:27,670] INFO Shutting down. (kafka.log.LogManager)
[2022-10-31 20:16:27,712] WARN /tmp/kraft-combined-logs/.kafka_cleanshutdown (kafka.utils.CoreUtils$)
java.nio.file.FileAlreadyExistsException: /tmp/kraft-combined-logs/.kafka_cleanshutdown
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:94)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
at java.base/java.nio.file.Files.newByteChannel(Files.java:371)
at java.base/java.nio.file.Files.createFile(Files.java:648)
at kafka.log.LogManager.$anonfun$shutdown$16(LogManager.scala:559)
at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:68)
at kafka.log.LogManager.$anonfun$shutdown$10(LogManager.scala:559)
at kafka.log.LogManager.$anonfun$shutdown$10$adapted(LogManager.scala:545)
at kafka.utils.Implicits$MapExtensionMethods$.$anonfun$forKeyValue$1(Implicits.scala:62)
at scala.collection.mutable.HashMap$Node.foreachEntry(HashMap.scala:633)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:499)
at kafka.log.LogManager.shutdown(LogManager.scala:545)
at kafka.server.BrokerServer.$anonfun$shutdown$21(BrokerServer.scala:519)
at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:68)
at kafka.server.BrokerServer.shutdown(BrokerServer.scala:519)
at kafka.server.BrokerServer.startup(BrokerServer.scala:443)
at kafka.server.KafkaRaftServer.$anonfun$startup$2(KafkaRaftServer.scala:114)
at kafka.server.KafkaRaftServer.$anonfun$startup$2$adapted(KafkaRaftServer.scala:114)
at scala.Option.foreach(Option.scala:437)
at kafka.server.KafkaRaftServer.startup(KafkaRaftServer.scala:114)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
[2022-10-31 20:16:27,716] INFO Shutdown complete. (kafka.log.LogManager)
[2022-10-31 20:16:27,717] INFO [ThrottledChannelReaper-Fetch]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,721] INFO [ThrottledChannelReaper-Fetch]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,722] INFO [ThrottledChannelReaper-Fetch]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,722] INFO [ThrottledChannelReaper-Produce]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,723] INFO [ThrottledChannelReaper-Produce]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,724] INFO [ThrottledChannelReaper-Produce]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,724] INFO [ThrottledChannelReaper-Request]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,725] INFO [ThrottledChannelReaper-Request]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,729] INFO [ThrottledChannelReaper-ControllerMutation]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,729] INFO [ThrottledChannelReaper-ControllerMutation]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,728] INFO [ThrottledChannelReaper-Request]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,730] INFO [ThrottledChannelReaper-ControllerMutation]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:27,733] INFO [SocketServer listenerType=BROKER, nodeId=1] Shutting down socket server (kafka.network.SocketServer)
[2022-10-31 20:16:27,808] INFO [SocketServer listenerType=BROKER, nodeId=1] Shutdown completed (kafka.network.SocketServer)
[2022-10-31 20:16:27,809] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics)
[2022-10-31 20:16:27,809] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics)
[2022-10-31 20:16:27,810] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics)
[2022-10-31 20:16:27,813] INFO Broker and topic stats closed (kafka.server.BrokerTopicStats)
[2022-10-31 20:16:27,814] INFO [BrokerLifecycleManager id=1] closed event queue. (org.apache.kafka.queue.KafkaEventQueue)
[2022-10-31 20:16:27,859] INFO App info kafka.server for 1 unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2022-10-31 20:16:27,860] INFO [BrokerServer id=1] shut down completed (kafka.server.BrokerServer)
[2022-10-31 20:16:27,860] INFO [BrokerServer id=1] Transition from SHUTTING_DOWN to SHUTDOWN (kafka.server.BrokerServer)
[2022-10-31 20:16:27,860] ERROR Exiting Kafka. (kafka.Kafka$)
[2022-10-31 20:16:27,862] INFO [kafka-raft-io-thread]: Shutting down (kafka.raft.KafkaRaftManager$RaftIoThread)
[2022-10-31 20:16:27,863] INFO [RaftManager nodeId=1] Beginning graceful shutdown (org.apache.kafka.raft.KafkaRaftClient)
[2022-10-31 20:16:28,111] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:28,111] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:28,711] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:28,711] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:29,191] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:29,191] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:29,626] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:29,626] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:30,134] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:30,134] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:30,691] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:30,692] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:31,298] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:31,298] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:31,917] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:31,917] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:32,426] INFO [RaftManager nodeId=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:32,426] WARN [RaftManager nodeId=1] Connection to node 2 (/10.20.10.5:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-10-31 20:16:32,867] WARN [RaftManager nodeId=1] Graceful shutdown timed out after 5000ms (org.apache.kafka.raft.KafkaRaftClient)
[2022-10-31 20:16:32,868] ERROR [kafka-raft-io-thread]: Graceful shutdown of RaftClient failed (kafka.raft.KafkaRaftManager$RaftIoThread)
java.util.concurrent.TimeoutException: Timeout expired before graceful shutdown completed
at org.apache.kafka.raft.KafkaRaftClient$GracefulShutdown.failWithTimeout(KafkaRaftClient.java:2411)
at org.apache.kafka.raft.KafkaRaftClient.maybeCompleteShutdown(KafkaRaftClient.java:2166)
at org.apache.kafka.raft.KafkaRaftClient.poll(KafkaRaftClient.java:2233)
at kafka.raft.KafkaRaftManager$RaftIoThread.doWork(RaftManager.scala:53)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
[2022-10-31 20:16:32,869] INFO [kafka-raft-io-thread]: Stopped (kafka.raft.KafkaRaftManager$RaftIoThread)
[2022-10-31 20:16:32,869] INFO [kafka-raft-io-thread]: Shutdown completed (kafka.raft.KafkaRaftManager$RaftIoThread)
[2022-10-31 20:16:32,891] INFO [kafka-raft-outbound-request-thread]: Shutting down (kafka.raft.RaftSendThread)
[2022-10-31 20:16:32,891] INFO [kafka-raft-outbound-request-thread]: Stopped (kafka.raft.RaftSendThread)
[2022-10-31 20:16:32,892] INFO [kafka-raft-outbound-request-thread]: Shutdown completed (kafka.raft.RaftSendThread)
[2022-10-31 20:16:32,903] INFO [ControllerServer id=1] shutting down (kafka.server.ControllerServer)
[2022-10-31 20:16:32,904] INFO [SocketServer listenerType=CONTROLLER, nodeId=1] Stopping socket server request processors (kafka.network.SocketServer)
[2022-10-31 20:16:32,926] INFO [SocketServer listenerType=CONTROLLER, nodeId=1] Stopped socket server request processors (kafka.network.SocketServer)
[2022-10-31 20:16:32,927] INFO [Controller 1] QuorumController#beginShutdown: shutting down event queue. (org.apache.kafka.queue.KafkaEventQueue)
[2022-10-31 20:16:32,927] INFO [SocketServer listenerType=CONTROLLER, nodeId=1] Shutting down socket server (kafka.network.SocketServer)
[2022-10-31 20:16:32,934] INFO [SocketServer listenerType=CONTROLLER, nodeId=1] Shutdown completed (kafka.network.SocketServer)
[2022-10-31 20:16:32,935] INFO [data-plane Kafka Request Handler on Broker 1], shutting down (kafka.server.KafkaRequestHandlerPool)
[2022-10-31 20:16:32,937] INFO [data-plane Kafka Request Handler on Broker 1], shut down completely (kafka.server.KafkaRequestHandlerPool)
[2022-10-31 20:16:32,938] INFO [ExpirationReaper-1-AlterAcls]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:32,938] INFO [ExpirationReaper-1-AlterAcls]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:32,939] INFO [ExpirationReaper-1-AlterAcls]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-10-31 20:16:32,940] INFO [ThrottledChannelReaper-Fetch]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,940] INFO [ThrottledChannelReaper-Fetch]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,940] INFO [ThrottledChannelReaper-Fetch]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,940] INFO [ThrottledChannelReaper-Produce]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,950] INFO [ThrottledChannelReaper-Produce]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,950] INFO [ThrottledChannelReaper-Produce]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,950] INFO [ThrottledChannelReaper-Request]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,951] INFO [ThrottledChannelReaper-Request]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,952] INFO [ThrottledChannelReaper-Request]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,952] INFO [ThrottledChannelReaper-ControllerMutation]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,953] INFO [ThrottledChannelReaper-ControllerMutation]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,953] INFO [ThrottledChannelReaper-ControllerMutation]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-10-31 20:16:32,953] INFO [Controller 1] closed event queue. (org.apache.kafka.queue.KafkaEventQueue)
[2022-10-31 20:16:32,956] INFO App info kafka.server for 1 unregistered (org.apache.kafka.common.utils.AppInfoParser)

samrat1004 · 1 November 2022 10:52

I am facing exactly same issue with 3 controller quorum voters.

There is one catch, if I race through to start all the three controllers in a few seconds, it works.

But starting one controller at a time with a few minutes apart always complain as below:
"Shutting down because we were unable to register with the controller quorum. "

As I reach to start the second controller, the first controller is down by that time…

ver-alex · 1 November 2022 12:19

How did you set up the listeners? Is there a separate listener for the controller on port 9093? Have you configured the controller.listener.names to list the dedicated controller listener defined in the listeners config?

fernandofariadev · 1 November 2022 16:58

Hello ver-alex

Thanks for your help.

Please see my server.properties

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@10.20.10.4:9093,2@10.20.10.5:9093
listeners=PLAINTEXT://10.20.10.4:9092,CONTROLLER://10.20.10.4:9093
inter.broker.listener.name=PLAINTEXT
advertised.listeners=PLAINTEXT://10.20.10.4:9092
controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
log.dirs=/tmp/server1/kraft-combined-logs

fernandofariadev · 4 November 2022 12:15

Hi, We managed to solve!

On first access, all nodes must be configured and active. After the first access, it votes the quorum and they can take down the other nodes.

faisalbasha19 · 31 August 2023 21:02

HI,

I am having the same issue, everytime I start the servers on all machines, its is throwing
Connection to node 2 (/172.16.24.33:9093) could not be established. Broker may not be available.

I am using ec2 instances, I have added the 9093 and 9092 on the security group just incase to allow the internal communications between brokers.

Please let me know the exact methods or steps to solve this particular issue.

Topic		Replies	Views
Apache Kafka 3.0 and Kraft quorum Ops	4	5801	23 September 2021
Proper way to horizontally scale Kafka 3 KRaft cluster Ops	0	20	3 December 2024
Kafka cluster collapsed for no reason Ops	0	15	30 November 2024
Can kafka-3.3.1 and kafka-0.10.0.1 coexist on one node? Cluster Replication	26	5218	4 November 2022
Deploy K@raft(KRAFT) mode multi-node kafka cluster for production Stream Processing	4	1896	19 January 2024

KAFKA cluster with kraft

Related topics