Unable to enable username password authentication for kafka server

Hi Team,

I am planning to enable authentication( username / password) for Kafka server.

i have followed below steps.

When i retsrat the kafka server its giving below error.

INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2022-07-27 08:35:46,417] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)

ANy suggestions please ?

or am i following correct approach to enable user name password authentication for Kafka Server

hey @Harijld

could you share some more details?

config and some more log information?

best,
michael

HI mmuehlbeyer,

Please find the detaisl…

zookeeper_jaas.conf file

Server {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="12345"
user_admin="12345";
};

added below parameters in zookeeper.properties file

#auth
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
jaasLoginRenew=3600000

executed below command

$ export KAFKA_OPTS=“-Djava.security.auth.login.config=/opt/kafka/config/zookeeper_jaas.conf”

Kafka -

kafka_server_jaas.conf

KafkaServer {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="admin"
  password="12345"
  user_admin="12345";
};

Client {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="admin"
  password="12345";
};

added below parameters in server.properties file

# AUTH

security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN

authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
allow.everyone.if.no.acl.found=true
listeners=SASL_PLAINTEXT://0.0.0.0:9092
advertised.listeners=SASL_PLAINTEXT://:9092

executed this step -

$ export KAFKA_OPTS=“-Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf”

restarted zookeeper and its working fine.
Restarted kafka server then its giving me below error.

[2022-07-27 08:35:46,118] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2022-07-27 08:35:46,406] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2022-07-27 08:35:46,417] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
java.lang.ClassNotFoundException: kafka.security.auth.AclAuthorizer
        at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.kafka.common.utils.Utils.loadClass(Utils.java:419)
        at org.apache.kafka.common.utils.Utils.newInstance(Utils.java:408)
        at kafka.security.authorizer.AuthorizerUtils$.createAuthorizer(AuthorizerUtils.scala:31)
        at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1658)
        at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1471)
        at kafka.Kafka$.buildServer(Kafka.scala:67)
        at kafka.Kafka$.main(Kafka.scala:87)
        at kafka.Kafka.main(Kafka.scala)


I am using 3.0.1 kafka version.

hey @Harijld

change

authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer

to

authorizer.class.name=kafka.security.authorizer.AclAuthorizer

Kafka 3.0 remove SimpleAclAuthorizer

hth,
michael

HI Michael,

i tried with below option but no luck.

# AUTH

security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN

authorizer.class.name=kafka.security.auth.AclAuthorizer
allow.everyone.if.no.acl.found=true
listeners=SASL_PLAINTEXT://10.119.37.108:9092
advertised.listeners=SASL_PLAINTEXT://10.119.37.108:9092

Error message -

[2022-07-27 14:50:15,407] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2022-07-27 14:50:15,695] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2022-07-27 14:50:15,707] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
java.lang.ClassNotFoundException: kafka.security.auth.AclAuthorizer
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.kafka.common.utils.Utils.loadClass(Utils.java:419)
at org.apache.kafka.common.utils.Utils.newInstance(Utils.java:408)
at kafka.security.authorizer.AuthorizerUtils$.createAuthorizer(AuthorizerUtils.scala:31)
at kafka.server.KafkaConfig.(KafkaConfig.scala:1658)
at kafka.server.KafkaConfig.(KafkaConfig.scala:1471)
at kafka.Kafka$.buildServer(Kafka.scala:67)
at kafka.Kafka$.main(Kafka.scala:87)
at kafka.Kafka.main(Kafka.scala)

I have executed below 2 commands after updating above mentioned parameters.

export KAFKA_OPTS=“-Djava.security.auth.login.config=/opt/kafka/config/zookeeper_jaas.conf”

export KAFKA_OPTS=“-Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf”

WIll these commands creating any problem ?

ok I see

classpath is set correctly
you’re using apache kafka not the confluent one, correct?

best,
michael

just to be sure

parameter looks like:

authorizer.class.name=kafka.security.authorizer.AclAuthorizer

right?
as the log is complaining about kafka.security.auth.AclAuthorizer

best,
michael

HI Michael,

Now old issue is resolved after changing the above line in server.property file.
Yes i am using apche kafka verison.

now getting new error.

[2022-07-27 16:33:13,637] INFO Completed load of Log(dir=/tmp/kafka-logs-new/applogmessages-0, topicId=iR3Hl7HhTwS-QRF5zNv40g, topic=applogmessages, partition=0, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 3ms (52/52 loaded in /tmp/kafka-logs-new) (kafka.log.LogManager)
[2022-07-27 16:33:13,638] INFO Loaded 52 logs in 340ms. (kafka.log.LogManager)
[2022-07-27 16:33:13,639] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2022-07-27 16:33:13,639] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2022-07-27 16:33:13,901] ERROR [KafkaServer id=18] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.IllegalArgumentException: Could not find a 'KafkaServer' or 'sasl_plaintext.KafkaServer' entry in the JAAS configuration. System property 'java.security.auth.login.config' is /opt/kafka/config/zookeeper_jaas.conf
        at org.apache.kafka.common.security.JaasContext.defaultContext(JaasContext.java:131)
        at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:96)
        at org.apache.kafka.common.security.JaasContext.loadServerContext(JaasContext.java:69)
        at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:168)
        at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)
        at kafka.server.BrokerToControllerChannelManagerImpl.newRequestThread(BrokerToControllerChannelManager.scala:189)
        at kafka.server.BrokerToControllerChannelManagerImpl.<init>(BrokerToControllerChannelManager.scala:168)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:284)
        at kafka.Kafka$.main(Kafka.scala:109)
        at kafka.Kafka.main(Kafka.scala)
[2022-07-27 16:33:13,903] INFO [KafkaServer id=18] shutting down (kafka.server.KafkaServer)
[2022-07-27 16:33:13,906] INFO Shutting down. (kafka.log.LogManager)
[2022-07-27 16:33:14,003] INFO Shutdown complete. (kafka.log.LogManager)
[2022-07-27 16:33:14,003] INFO [feature-zk-node-event-process-thread]: Shutting down (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2022-07-27 16:33:14,003] INFO [feature-zk-node-event-process-thread]: Stopped (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2022-07-27 16:33:14,003] INFO [feature-zk-node-event-process-thread]: Shutdown completed (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2022-07-27 16:33:14,004] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2022-07-27 16:33:14,113] INFO Session: 0x100cd21b5650001 closed (org.apache.zookeeper.ZooKeeper)
[2022-07-27 16:33:14,113] INFO EventThread shut down for session: 0x100cd21b5650001 (org.apache.zookeeper.ClientCnxn)

Hi Michel,

One more observation…

I am running both zookeeper and Kafka server on same machine.

if i execute below command and then if i start the kafka server, getting below error
export KAFKA_OPTS="—Djava.security.auth.login.config=/opt/kafka/config/zookeeper_jaas.conf

Error

[2022-07-27 16:59:34,898] ERROR [KafkaServer id=18] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.IllegalArgumentException: Could not find a 'KafkaServer' or 'sasl_plaintext.KafkaServer' entry in the JAAS configuration. System property 'java.security.auth.login.config' is /opt/kafka/config/zookeeper_jaas.conf
        at org.apache.kafka.common.security.JaasContext.defaultContext(JaasContext.java:131)
        at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:96)
        at org.apache.kafka.common.security.JaasContext.loadServerContext(JaasContext.java:69)
        at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:168)
        at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)
        at kafka.server.BrokerToControllerChannelManagerImpl.newRequestThread(BrokerToControllerChannelManager.scala:189)
        at kafka.server.BrokerToControllerChannelManagerImpl.<init>(BrokerToControllerChannelManager.scala:168)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:284)
        at kafka.Kafka$.main(Kafka.scala:109)
        at kafka.Kafka.main(Kafka.scala)
[2022-07-27 16:59:34,900] INFO [KafkaServer id=18] shutting down (kafka.server.KafkaServer)

And if set below value , and start the kafka server, getting below error.

export KAFKA_OPTS=“-Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf”

error message -

[2022-07-27 17:01:02,376] ERROR SASL authentication failed using login context 'Client'. (org.apache.zookeeper.client.ZooKeeperSaslClient)
javax.security.sasl.SaslException: Error in authenticating with a Zookeeper Quorum member: the quorum member's saslToken is null.


[2022-07-27 17:01:02,383] INFO EventThread shut down for session: 0x100cd40db240001 (org.apache.zookeeper.ClientCnxn)
[2022-07-27 17:01:02,438] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /consumers
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at kafka.zookeeper.AsyncResponse.maybeThrow(ZooKeeperClient.scala:566)

How to set both the values in the same server…

export KAFKA_OPTS=“-Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf”
export KAFKA_OPTS=“-Djava.security.auth.login.config=/opt/kafka/config/zookeeper_jaas.conf”

hey @Harijld

I think the easiest way is to use 2 separate shell sessions.

or use 2 separate user if you’d like to separate in this way.

best,
michael

HI Michael,

I ran below 2 commands with .sh and now it is running but giving error while connecting to zookeeper.

export KAFKA_OPTS=-Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf
./bin/kafka-server-start.sh config/server.properties



export KAFKA_OPTS=-Djava.security.auth.login.config=/opt/kafka/config/zookeeper_jaas.conf
./bin/zookeeper-server-start.sh config/zookeeper.properties

Error message -

[2022-07-28 13:20:12,213] INFO [SocketServer listenerType=ZK_BROKER, nodeId=18] Failed authentication with /10.119.37.107 (channelId=10.xxx.xx.xxx:9092-10.xxx.xx.xxx:44958-3) (Unexpected Kafka request of type METADATA during SASL handshake.) (org.apache.kafka.common.network.Selector)
[2022-07-28 13:20:12,618] INFO [SocketServer listenerType=ZK_BROKER, nodeId=18] Failed authentication with /10.xxx.xx.xxx (channelId=10.xxx.xx.xxx:9092-10.xxx.xx.xxx:44960-3) (Unexpected Kafka request of type METADATA during SASL handshake.) (org.apache.kafka.common.network.Selector)

Some property / parameter issue at client or server end.

Hello. Reminder that this section of the forum is for the Kafka Connect API, not general topics of “connecting to Kafka”.

For that, Ops or Clients sections would be preferred.