Hello,
i dont have previous experience configuring Kraft and im trying to set up Kafka cluster using Kraft in combined mode.
My goal is to achieve kafka servers to authenticate via Kerboros (SASL_PLAINTEXT protocol).
I followed official Confluent documentation, such as
https://docs.confluent.io/platform/current/kafka/authentication_sasl/authentication_sasl_gssapi.html
Im stuck at the state, where my kafka service can start, it communicates with other controllers and looks stable. But issue with functionality appears when i want to contact brokers using commands such as:
/usr/bin/kafka-topics --list --bootstrap-server hostname
or
/usr/bin/kafka-metadata-quorum --bootstrap-server hostname:9092 describe --status
In /var/log/kafka/server.log these messages repeat until the command times out (first octets of IP addresses replaced with X.X.X):
DEBUG Accepted connection from /X.X.X.116:58312 on /X.X.X.117:9092 and assigned it to processor 10, sendBufferSize [actual|requested]: [102400|102400] recvBufferSize [actual|requested]: [102400|102400] (kafka.network.DataPlaneAcceptor)
DEBUG Processor 10 listening to new connection from /X.X.X.116:58312 (kafka.network.Processor)
INFO [SocketServer listenerType=BROKER, nodeId=1] Failed authentication with /X.X.X.116 (channelId=X.X.X.117:9092-X.X.X.116:58312-0) (Unexpected Kafka request of type METADATA during SASL handshake.) (org.apache.kafka.common.network.Selector)
I tried many things, but nothing worked and im out of ideas. I would be very grateful for advices.
Here are details of my kafka cluster:
/usr/lib/systemd/system/confluent-kafka.service
[Unit]
Description=Apache Kafka - broker
Documentation=http://docs.confluent.io/
After=network.target confluent-zookeeper.target[Service]
Type=simple
User=cp-kafka
Group=confluent
ExecStart=/usr/bin/kafka-server-start /etc/kafka/kraft/server.properties
Environment=“KAFKA_OPTS=-Dsun.security.krb5.debug=true -Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/etc/kafka/kraft/kafka_server_jaas.conf”
LimitNOFILE=1000000
TimeoutStopSec=180
Restart=no[Install]
WantedBy=multi-user.target
However the kerb debug option (-Dsun.security.krb5.debug=true) doesnt seem to be working. I couldnt see no new messages in /var/log/kafka/server.log
/etc/kafka/kraft/server.properties (actual hostnames replaced with generic hostname names)
process.roles=broker,controller
node.id=1
controller.quorum.voters=1@hostname1:9093,2@hostname2.oskarmobil.cz:9093,3@hostname3:9093listeners=SASL_PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
advertised.listeners=SASL_PLAINTEXT://hostname1:9092
controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:SASL_PLAINTEXT,SASL_PLAINTEXT:SASL_PLAINTEXTnum.network.threads=16
num.io.threads=12
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600metadata.log.dir=/var/lib/kraft
log.dirs=/mnt/disk-1/kafkanum.partitions=12
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2
log.retention.hours=336
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
group.initial.rebalance.delay.ms=0confluent.license.topic.replication.factor=1
confluent.metadata.topic.replication.factor=1
confluent.security.event.logger.exporter.kafka.topic.replicas=1delete.topic.enable=true
auto.create.topics.enable=falsesasl.enabled.mechanisms=GSSAPI
sasl.mechanism.inter.broker.protocol=GSSAPI
sasl.mechanism.controller.protocol=GSSAPIsecurity.inter.broker.protocol=SASL_PLAINTEXT
security.protocol=SASL_PLAINTEXTlistener.name.sasl_plaintext.gssapi.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab=“/etc/security/keytabs/hostname1.keytab”
principal=“kafka/hostname1@REALM”;sasl.kerberos.service.name=kafka
/etc/kafka/kraft/kafka_server_jaas.conf
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab=“/etc/security/keytabs/hostname1.keytab”
principal=“kafka/hostname1@REALM”;
};
I wasnt actually planning on using dedicated jaas file as confluent docs literally says “While use of separate JAAS files is supported, it is not the recommended approach. Instead, use the listener configuration specified…”
But if i followed the docs, the kafka wouldnt start at all, with error that KafkaServer is not defined and i need to specify jaas conf file. So i created the jaas file with basically same Krb5LoginModule.
I can provide additional details/configs if needed.
One thing i noticed that i couldnt wrap my head around is that controllers seems to be able to authenticate, but brokers cant…
As seen in the /var/log/kafka/server.log:
DEBUG Accepted connection from /X.X.X.117:46588 on /X.X.X.117:9093 and assigned it to processor 10, sendBufferSize [actual|requested]: [102400|102400] recvBufferSize [actual|requested]: [102400|102400] (kafka.network.DataPlaneAcceptor)
DEBUG Processor 10 listening to new connection from /X.X.X.117:46588 (kafka.network.Processor)
DEBUG Accepted connection from /X.X.X.116:36048 on /X.X.X.117:9093 and assigned it to processor 11, sendBufferSize [actual|requested]: [102400|102400] recvBufferSize [actual|requested]: [102400|102400] (kafka.network.DataPlaneAcceptor)
DEBUG Processor 11 listening to new connection from /X.X.X.116:36048 (kafka.network.Processor)
INFO Successfully authenticated client: authenticationID=kafka/hostname1@REALM; authorizationID=kafka/hostname1@REALM. (org.apache.kafka.common.security.authenticator.SaslServerCallbackHandler)
INFO Successfully authenticated client: authenticationID=kafka/hostname2@REALM; authorizationID=kafka/hostname2@REALM. (org.apache.kafka.common.security.authenticator.SaslServerCallbackHandler)
Thank you for your time.
Regard, Lukas
