Unable to enable username password authentication for kafka server

Harijld · 27 July 2022 07:48

Hi Team,

I am planning to enable authentication( username / password) for Kafka server.

i have followed below steps.

When i retsrat the kafka server its giving below error.

INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2022-07-27 08:35:46,417] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)

ANy suggestions please ?

or am i following correct approach to enable user name password authentication for Kafka Server

mmuehlbeyer · 27 July 2022 09:21

hey @Harijld

could you share some more details?

config and some more log information?

best,
michael

Harijld · 27 July 2022 09:31

HI mmuehlbeyer,

Please find the detaisl…

zookeeper_jaas.conf file

Server {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="12345"
user_admin="12345";
};

added below parameters in zookeeper.properties file

#auth
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
jaasLoginRenew=3600000

executed below command

$ export KAFKA_OPTS=“-Djava.security.auth.login.config=/opt/kafka/config/zookeeper_jaas.conf”

Kafka -

kafka_server_jaas.conf

KafkaServer {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="admin"
  password="12345"
  user_admin="12345";
};

Client {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="admin"
  password="12345";
};

added below parameters in server.properties file

# AUTH

security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN

authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
allow.everyone.if.no.acl.found=true
listeners=SASL_PLAINTEXT://0.0.0.0:9092
advertised.listeners=SASL_PLAINTEXT://:9092

executed this step -

$ export KAFKA_OPTS=“-Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf”

restarted zookeeper and its working fine.
Restarted kafka server then its giving me below error.

[2022-07-27 08:35:46,118] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2022-07-27 08:35:46,406] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2022-07-27 08:35:46,417] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
java.lang.ClassNotFoundException: kafka.security.auth.AclAuthorizer
        at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.kafka.common.utils.Utils.loadClass(Utils.java:419)
        at org.apache.kafka.common.utils.Utils.newInstance(Utils.java:408)
        at kafka.security.authorizer.AuthorizerUtils$.createAuthorizer(AuthorizerUtils.scala:31)
        at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1658)
        at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1471)
        at kafka.Kafka$.buildServer(Kafka.scala:67)
        at kafka.Kafka$.main(Kafka.scala:87)
        at kafka.Kafka.main(Kafka.scala)

I am using 3.0.1 kafka version.

mmuehlbeyer · 27 July 2022 09:38

hey @Harijld

change

authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer

to

authorizer.class.name=kafka.security.authorizer.AclAuthorizer

Kafka 3.0 remove SimpleAclAuthorizer

hth,
michael

Harijld · 27 July 2022 10:51

HI Michael,

i tried with below option but no luck.

# AUTH

security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN

authorizer.class.name=kafka.security.auth.AclAuthorizer
allow.everyone.if.no.acl.found=true
listeners=SASL_PLAINTEXT://10.119.37.108:9092
advertised.listeners=SASL_PLAINTEXT://10.119.37.108:9092

Error message -

[2022-07-27 14:50:15,407] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2022-07-27 14:50:15,695] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2022-07-27 14:50:15,707] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
java.lang.ClassNotFoundException: kafka.security.auth.AclAuthorizer
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.kafka.common.utils.Utils.loadClass(Utils.java:419)
at org.apache.kafka.common.utils.Utils.newInstance(Utils.java:408)
at kafka.security.authorizer.AuthorizerUtils$.createAuthorizer(AuthorizerUtils.scala:31)
at kafka.server.KafkaConfig.(KafkaConfig.scala:1658)
at kafka.server.KafkaConfig.(KafkaConfig.scala:1471)
at kafka.Kafka$.buildServer(Kafka.scala:67)
at kafka.Kafka$.main(Kafka.scala:87)
at kafka.Kafka.main(Kafka.scala)

Harijld · 27 July 2022 10:53

I have executed below 2 commands after updating above mentioned parameters.

export KAFKA_OPTS=“-Djava.security.auth.login.config=/opt/kafka/config/zookeeper_jaas.conf”

export KAFKA_OPTS=“-Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf”

WIll these commands creating any problem ?

mmuehlbeyer · 27 July 2022 11:04

ok I see

classpath is set correctly
you’re using apache kafka not the confluent one, correct?

best,
michael

mmuehlbeyer · 27 July 2022 11:15

just to be sure

parameter looks like:

authorizer.class.name=kafka.security.authorizer.AclAuthorizer

right?
as the log is complaining about kafka.security.auth.AclAuthorizer

best,
michael

Harijld · 27 July 2022 12:36

HI Michael,

Now old issue is resolved after changing the above line in server.property file.
Yes i am using apche kafka verison.

now getting new error.

[2022-07-27 16:33:13,637] INFO Completed load of Log(dir=/tmp/kafka-logs-new/applogmessages-0, topicId=iR3Hl7HhTwS-QRF5zNv40g, topic=applogmessages, partition=0, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 3ms (52/52 loaded in /tmp/kafka-logs-new) (kafka.log.LogManager)
[2022-07-27 16:33:13,638] INFO Loaded 52 logs in 340ms. (kafka.log.LogManager)
[2022-07-27 16:33:13,639] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2022-07-27 16:33:13,639] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2022-07-27 16:33:13,901] ERROR [KafkaServer id=18] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.IllegalArgumentException: Could not find a 'KafkaServer' or 'sasl_plaintext.KafkaServer' entry in the JAAS configuration. System property 'java.security.auth.login.config' is /opt/kafka/config/zookeeper_jaas.conf
        at org.apache.kafka.common.security.JaasContext.defaultContext(JaasContext.java:131)
        at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:96)
        at org.apache.kafka.common.security.JaasContext.loadServerContext(JaasContext.java:69)
        at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:168)
        at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)
        at kafka.server.BrokerToControllerChannelManagerImpl.newRequestThread(BrokerToControllerChannelManager.scala:189)
        at kafka.server.BrokerToControllerChannelManagerImpl.<init>(BrokerToControllerChannelManager.scala:168)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:284)
        at kafka.Kafka$.main(Kafka.scala:109)
        at kafka.Kafka.main(Kafka.scala)
[2022-07-27 16:33:13,903] INFO [KafkaServer id=18] shutting down (kafka.server.KafkaServer)
[2022-07-27 16:33:13,906] INFO Shutting down. (kafka.log.LogManager)
[2022-07-27 16:33:14,003] INFO Shutdown complete. (kafka.log.LogManager)
[2022-07-27 16:33:14,003] INFO [feature-zk-node-event-process-thread]: Shutting down (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2022-07-27 16:33:14,003] INFO [feature-zk-node-event-process-thread]: Stopped (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2022-07-27 16:33:14,003] INFO [feature-zk-node-event-process-thread]: Shutdown completed (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2022-07-27 16:33:14,004] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2022-07-27 16:33:14,113] INFO Session: 0x100cd21b5650001 closed (org.apache.zookeeper.ZooKeeper)
[2022-07-27 16:33:14,113] INFO EventThread shut down for session: 0x100cd21b5650001 (org.apache.zookeeper.ClientCnxn)

Harijld · 27 July 2022 13:04

Hi Michel,

One more observation…

I am running both zookeeper and Kafka server on same machine.

if i execute below command and then if i start the kafka server, getting below error
export KAFKA_OPTS="—Djava.security.auth.login.config=/opt/kafka/config/zookeeper_jaas.conf

Error

[2022-07-27 16:59:34,898] ERROR [KafkaServer id=18] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.IllegalArgumentException: Could not find a 'KafkaServer' or 'sasl_plaintext.KafkaServer' entry in the JAAS configuration. System property 'java.security.auth.login.config' is /opt/kafka/config/zookeeper_jaas.conf
        at org.apache.kafka.common.security.JaasContext.defaultContext(JaasContext.java:131)
        at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:96)
        at org.apache.kafka.common.security.JaasContext.loadServerContext(JaasContext.java:69)
        at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:168)
        at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)
        at kafka.server.BrokerToControllerChannelManagerImpl.newRequestThread(BrokerToControllerChannelManager.scala:189)
        at kafka.server.BrokerToControllerChannelManagerImpl.<init>(BrokerToControllerChannelManager.scala:168)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:284)
        at kafka.Kafka$.main(Kafka.scala:109)
        at kafka.Kafka.main(Kafka.scala)
[2022-07-27 16:59:34,900] INFO [KafkaServer id=18] shutting down (kafka.server.KafkaServer)

And if set below value , and start the kafka server, getting below error.

export KAFKA_OPTS=“-Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf”

error message -

[2022-07-27 17:01:02,376] ERROR SASL authentication failed using login context 'Client'. (org.apache.zookeeper.client.ZooKeeperSaslClient)
javax.security.sasl.SaslException: Error in authenticating with a Zookeeper Quorum member: the quorum member's saslToken is null.


[2022-07-27 17:01:02,383] INFO EventThread shut down for session: 0x100cd40db240001 (org.apache.zookeeper.ClientCnxn)
[2022-07-27 17:01:02,438] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /consumers
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at kafka.zookeeper.AsyncResponse.maybeThrow(ZooKeeperClient.scala:566)

How to set both the values in the same server…

export KAFKA_OPTS=“-Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf”
export KAFKA_OPTS=“-Djava.security.auth.login.config=/opt/kafka/config/zookeeper_jaas.conf”

mmuehlbeyer · 28 July 2022 05:43

hey @Harijld

I think the easiest way is to use 2 separate shell sessions.

or use 2 separate user if you’d like to separate in this way.

best,
michael

Harijld · 28 July 2022 09:24

HI Michael,

I ran below 2 commands with .sh and now it is running but giving error while connecting to zookeeper.

export KAFKA_OPTS=-Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf
./bin/kafka-server-start.sh config/server.properties



export KAFKA_OPTS=-Djava.security.auth.login.config=/opt/kafka/config/zookeeper_jaas.conf
./bin/zookeeper-server-start.sh config/zookeeper.properties

Error message -

[2022-07-28 13:20:12,213] INFO [SocketServer listenerType=ZK_BROKER, nodeId=18] Failed authentication with /10.119.37.107 (channelId=10.xxx.xx.xxx:9092-10.xxx.xx.xxx:44958-3) (Unexpected Kafka request of type METADATA during SASL handshake.) (org.apache.kafka.common.network.Selector)
[2022-07-28 13:20:12,618] INFO [SocketServer listenerType=ZK_BROKER, nodeId=18] Failed authentication with /10.xxx.xx.xxx (channelId=10.xxx.xx.xxx:9092-10.xxx.xx.xxx:44960-3) (Unexpected Kafka request of type METADATA during SASL handshake.) (org.apache.kafka.common.network.Selector)

Some property / parameter issue at client or server end.

OneCricketeer · 29 July 2022 16:46

Hello. Reminder that this section of the forum is for the Kafka Connect API, not general topics of “connecting to Kafka”.

For that, Ops or Clients sections would be preferred.

system · 28 August 2022 16:46

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
How to change user in Zookeeper client Clients	0	2142	31 May 2023
Plain SASL set up is not working Clients	2	6071	29 September 2022
Connection between Zookeeper and Kafka server doesnt working Kafka Streams	0	1118	18 December 2023
SASL/PLAIN Authentication Ops	3	1018	8 January 2024
Failed authentication due to: SSL handshake failed Ops	0	10220	20 December 2021

Unable to enable username password authentication for kafka server

Related topics