Hello, I’m trying to setup a Kafka Kraft cluster with 3 nodes, each acting as a controller and broker at the same time. Configuring the controller listener to plaintext instead of sasl_ssl let’s the cluster initialize without any problems. But I want all communication to be encrypted and all brokers/controllers to authenticate themselves.
When i start up my cluster I’m receiving following errors:
Dec 19 14:43:49 kafka01 kafka-server-start.sh[947588]: 2023-12-19 14:43:49,271 ERROR [kafka-1-raft-io-thread:Logging@76] - [kafka-1-raft-io-thread]: Graceful shutdown of RaftClient failed
Dec 19 14:43:49 kafka01 kafka-server-start.sh[947588]: 2023-12-19 14:43:49,301 ERROR [kafka-1-metadata-loader-event-handler:StandardAuthorizer@96] - [StandardAuthorizer 1] Failed to complete initial ACL load process.
And the entire log is filled with this message:
Dec 19 14:33:05 kafka03 kafka-server-start.sh[471117]: 2023-12-19 14:33:05,303 INFO [kafka-3-metadata-loader-event-handler:MetadataLoader@234] - [MetadataLoader id=3] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet.
Dec 19 14:33:05 kafka03 kafka-server-start.sh[471117]: 2023-12-19 14:33:05,404 INFO [kafka-3-metadata-loader-event-handler:MetadataLoader@234] - [MetadataLoader id=3] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet.
Dec 19 14:33:05 kafka03 kafka-server-start.sh[471117]: 2023-12-19 14:33:05,506 INFO [kafka-3-metadata-loader-event-handler:MetadataLoader@234] - [MetadataLoader id=3] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet.
Dec 19 14:33:05 kafka03 kafka-server-start.sh[471117]: 2023-12-19 14:33:05,607 INFO [kafka-3-metadata-loader-event-handler:MetadataLoader@234] - [MetadataLoader id=3] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet.
Dec 19 14:33:05 kafka03 kafka-server-start.sh[471117]: 2023-12-19 14:33:05,707 INFO [kafka-3-metadata-loader-event-handler:MetadataLoader@234] - [MetadataLoader id=3] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet.
Dec 19 14:33:05 kafka03 kafka-server-start.sh[471117]: 2023-12-19 14:33:05,807 INFO [kafka-3-metadata-loader-event-handler:MetadataLoader@234] - [MetadataLoader id=3] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet.
Dec 19 14:33:05 kafka03 kafka-server-start.sh[471117]: 2023-12-19 14:33:05,915 INFO [kafka-3-metadata-loader-event-handler:MetadataLoader@234] - [MetadataLoader id=3] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet.
Dec 19 14:33:06 kafka03 kafka-server-start.sh[471117]: 2023-12-19 14:33:06,016 INFO [kafka-3-metadata-loader-event-handler:MetadataLoader@234] - [MetadataLoader id=3] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet.
This is my configuration:
log.dirs=/data/kafka
process.roles=broker,controller
controller.quorum.voters=1@kafka01:9093,2@kafka02:9093,3@kafka03:9093
controller.listener.names=CONTROLLER
listeners=OPERATOR://:9092,CONTROLLER://:9093,CLIENT://:9094
advertised.listeners=OPERATOR://:9092,CLIENT://:9094
listener.security.protocol.map=OPERATOR:SASL_SSL,CONTROLLER:SASL_PLAINTEXT,CLIENT:SASL_SSL
inter.broker.listener.name=OPERATOR
sasl.enabled.mechanisms=PLAIN
sasl.mechanism.controller.protocol=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN
ssl.keystore.location=/etc/kafka.d/server.keystore.jks
ssl.keystore.password=[password]
ssl.key.password=[password]
ssl.truststore.location=/etc/kafka.d/server.truststore.jks
ssl.truststore.password=[password]
ssl.client.auth=required
ssl.endpoint.identification.algorithm=HTTPS
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
super.users=User:broker;User:operator;User:operator;
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
num.network.threads=3
group.initial.rebalance.delay.ms=0
auto.create.topics.enable=true
num.partitions=1
num.io.threads=8
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.bytes = -1
log.retention.hours = -1
log.retention.minutes = -1
log.retention.ms = -1
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
broker.id=1
node.id=1
I’m not sure where to look for the error because the server is running, but the controller isn’t ready yet.