Everything is fine when i enabled TLS Auth. Today i try to enable SASL Auth at Kraft mode, but after a few hour working i stuck at this error Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256.
This is my server configuration
apiVersion: v1
kind: ConfigMap
metadata:
name: kafka-config
namespace: kafka
data:
server.properties: |
# ... default configuration
############################# Group Coordinator Settings #############################
# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
# The default value for this is 3 seconds.
# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.
# group.initial.rebalance.delay.ms=0
message.max.bytes=1000012
auto.create.topics.enable=true
delete.topic.enable=false
default.replication.factor=1
inter.broker.listener.name=BROKER
# remember to configure protocol correctly, otherwise it will raise `No serviceName defined in either JAAS or Kafka config` error
sasl.enabled.mechanisms=SCRAM-SHA-256
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256
sasl.mechanism.controller.protocol=SCRAM-SHA-256
############################# ACL #############################
allow.everyone.if.no.acl.found=true
# authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
super.users=User:admin,User:interbroker,User:controller
############################# SSL #############################
# Server host name verification may be disabled by setting ssl.endpoint.identification.algorithm to an empty string.
ssl.endpoint.identification.algorithm=https
ssl.client.auth=requested
ssl.keystore.type=PEM
ssl.truststore.type=PEM
ssl.truststore.location=/etc/kafka/certs/truststore.pem
ssl.keystore.location=/etc/kafka/certs/keystore.pem
############################# sasl jaas config #############################
# listener.name.<listener-name>.<sasl.mechanism.inter.broker.protocol>.sasl.jaas.config
listener.name.inter_broker.scram-sha-256.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="zuMCS0l0m8LPMr8m" user_interbroker="k3o2hAtv1B5Q4KPP";
listener.name.controller.scram-sha-256.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="controller" password="ePMgJmv34x3KufGB" user_controller="ePMgJmv34x3KufGB";
############################# My Config #############################
kafka_jaas.conf: |
KafkaServer {
org.apache.kafka.common.security.scram.ScramLoginModule required
username="admin"
password="zuMCS0l0m8LPMr8m"
user_interbroker="k3o2hAtv1B5Q4KPP"
user_controller="ePMgJmv34x3KufGB"
user_client="AlhUFaGt3o8GTz9f";
};
Container Envs
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: KAFKA_PROCESS_ROLES
value: broker,controller
- name: KAFKA_LISTENERS
# {LISTENER_NAME}://{hostname}:{port}
value: BROKER://:9092,CONTROLLER://:9093,INTER_BROKER://:9094
- name: KAFKA_ADVERTISED_LISTENERS
value: BROKER://:9092,INTER_BROKER://:9094
- name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
# PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL
value: BROKER:SASL_SSL,INTER_BROKER:SASL_SSL,CONTROLLER:SASL_SSL
- name: KAFKA_CONTROLLER_QUORUM_VOTERS
value: 0@kafka-0.kafka-headless.kafka.svc.cluster.local:9093,1@kafka-1.kafka-headless.kafka.svc.cluster.local:9093,2@kafka-2.kafka-headless.kafka.svc.cluster.local:9093
- name: KAFKA_INTER_BROKER_LISTENER_NAME
# inter.broker.listener.name must be a listener name defined in advertised.listeners
value: INTER_BROKER
- name: KAFKA_CONTROLLER_LISTENER_NAMES
value: CONTROLLER
- name: CLUSTER_ID
value: MkU3OEVBNTcwNTJENDM2Qk
- name: KAFKA_JMX_PORT
value: "9101"
- name: KAFKA_JMX_HOSTNAME
value: localhost
- name: KAFKA_HEAP_OPTS
value: "-Xms1g -Xmx1g" # suggest xms6g xmx6g
- name: KAFKA_GC_LOG_OPTS
value: "-XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80"
- name: KAFKA_OPTS
value: -Djava.security.auth.login.config=/etc/kafka/kraft/kafka_jaas.conf
- name: KAFKA_LOG_DIRS
value: /var/log/kafka/
I can’t figure out why it passes invalid credentials. Can anyone tell me.
More error details
[2023-07-25 13:07:05,432] ERROR [RaftManager nodeId=0] Connection to node 2 (kafka-2.kafka-headless.kafka.svc.cluster.local/10.244.5.196:9093) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
[2023-07-25 13:07:05,434] ERROR [kafka-raft-outbound-request-thread]: Failed to send the following request due to authentication error: ClientRequest(expectResponse=true, callback=kafka.raft.KafkaNetworkChannel$$Lambda$627/0x0000000100551440@58b5a1d, destination=2, correlationId=518, clientId=raft-client-0, createdTimeMs=1690290424955, requestBuilder=VoteRequestData(clusterId='MkU3OEVBNTcwNTJENDM2Qg', topics=[TopicData(topicName='__cluster_metadata', partitions=[PartitionData(partitionIndex=0, candidateEpoch=12, candidateId=0, lastOffsetEpoch=0, lastOffset=0)])])) (kafka.raft.RaftSendThread)
[2023-07-25 13:07:05,435] ERROR Request OutboundRequest(correlationId=518, data=VoteRequestData(clusterId='MkU3OEVBNTcwNTJENDM2Qg', topics=[TopicData(topicName='__cluster_metadata', partitions=[PartitionData(partitionIndex=0, candidateEpoch=12, candidateId=0, lastOffsetEpoch=0, lastOffset=0)])]), createdTimeMs=1690290424955, destinationId=2) failed due to authentication error (kafka.raft.KafkaNetworkChannel)
org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256
[2023-07-25 13:07:05,435] ERROR [RaftManager nodeId=0] Unexpected error NETWORK_EXCEPTION in VOTE response: InboundResponse(correlationId=518, data=VoteResponseData(errorCode=13, topics=[]), sourceId=2) (org.apache.kafka.raft.KafkaRaftClient)
[2023-07-25 13:07:05,471] INFO [SocketServer listenerType=CONTROLLER, nodeId=0] Failed authentication with /10.244.5.196 (channelId=10.244.5.194:9093-10.244.5.196:41032-10) (Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256) (org.apache.kafka.common.network.Selector)