Got `Authentication failed` even i already pass correct username and password

Everything is fine when i enabled TLS Auth. Today i try to enable SASL Auth at Kraft mode, but after a few hour working i stuck at this error Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256.

This is my server configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: kafka-config
  namespace: kafka
data:
  server.properties: |
    # ... default configuration

    ############################# Group Coordinator Settings #############################

    # The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
    # The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
    # The default value for this is 3 seconds.
    # We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
    # However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.

    # group.initial.rebalance.delay.ms=0
    message.max.bytes=1000012

    auto.create.topics.enable=true
    delete.topic.enable=false
    default.replication.factor=1

    inter.broker.listener.name=BROKER

    # remember to configure protocol correctly, otherwise it will raise `No serviceName defined in either JAAS or Kafka config` error
    sasl.enabled.mechanisms=SCRAM-SHA-256
    sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256
    sasl.mechanism.controller.protocol=SCRAM-SHA-256

    ############################# ACL #############################

    allow.everyone.if.no.acl.found=true
    # authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
    super.users=User:admin,User:interbroker,User:controller

    ############################# SSL #############################

    # Server host name verification may be disabled by setting ssl.endpoint.identification.algorithm to an empty string.
    ssl.endpoint.identification.algorithm=https
    ssl.client.auth=requested

    ssl.keystore.type=PEM
    ssl.truststore.type=PEM
    ssl.truststore.location=/etc/kafka/certs/truststore.pem
    ssl.keystore.location=/etc/kafka/certs/keystore.pem

    ############################# sasl jaas config #############################

    # listener.name.<listener-name>.<sasl.mechanism.inter.broker.protocol>.sasl.jaas.config

    listener.name.inter_broker.scram-sha-256.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="zuMCS0l0m8LPMr8m" user_interbroker="k3o2hAtv1B5Q4KPP";

    listener.name.controller.scram-sha-256.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="controller" password="ePMgJmv34x3KufGB" user_controller="ePMgJmv34x3KufGB";

    ############################# My Config #############################

  kafka_jaas.conf: |
    KafkaServer {
      org.apache.kafka.common.security.scram.ScramLoginModule required
      username="admin"
      password="zuMCS0l0m8LPMr8m"
      user_interbroker="k3o2hAtv1B5Q4KPP"
      user_controller="ePMgJmv34x3KufGB"
      user_client="AlhUFaGt3o8GTz9f";
    };

Container Envs

            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: KAFKA_PROCESS_ROLES
              value: broker,controller
            - name: KAFKA_LISTENERS
              # {LISTENER_NAME}://{hostname}:{port}
              value: BROKER://:9092,CONTROLLER://:9093,INTER_BROKER://:9094
            - name: KAFKA_ADVERTISED_LISTENERS
              value: BROKER://:9092,INTER_BROKER://:9094
            - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
              # PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL
              value: BROKER:SASL_SSL,INTER_BROKER:SASL_SSL,CONTROLLER:SASL_SSL
            - name: KAFKA_CONTROLLER_QUORUM_VOTERS
              value: 0@kafka-0.kafka-headless.kafka.svc.cluster.local:9093,1@kafka-1.kafka-headless.kafka.svc.cluster.local:9093,2@kafka-2.kafka-headless.kafka.svc.cluster.local:9093
            - name: KAFKA_INTER_BROKER_LISTENER_NAME
              # inter.broker.listener.name must be a listener name defined in advertised.listeners
              value: INTER_BROKER
            - name: KAFKA_CONTROLLER_LISTENER_NAMES
              value: CONTROLLER
            - name: CLUSTER_ID
              value: MkU3OEVBNTcwNTJENDM2Qk
            - name: KAFKA_JMX_PORT
              value: "9101"
            - name: KAFKA_JMX_HOSTNAME
              value: localhost
            - name: KAFKA_HEAP_OPTS
              value: "-Xms1g -Xmx1g" # suggest xms6g xmx6g
            - name: KAFKA_GC_LOG_OPTS
              value: "-XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80"
            - name: KAFKA_OPTS
              value: -Djava.security.auth.login.config=/etc/kafka/kraft/kafka_jaas.conf
            - name: KAFKA_LOG_DIRS
              value: /var/log/kafka/

I can’t figure out why it passes invalid credentials. Can anyone tell me.

More error details

[2023-07-25 13:07:05,432] ERROR [RaftManager nodeId=0] Connection to node 2 (kafka-2.kafka-headless.kafka.svc.cluster.local/10.244.5.196:9093) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
[2023-07-25 13:07:05,434] ERROR [kafka-raft-outbound-request-thread]: Failed to send the following request due to authentication error: ClientRequest(expectResponse=true, callback=kafka.raft.KafkaNetworkChannel$$Lambda$627/0x0000000100551440@58b5a1d, destination=2, correlationId=518, clientId=raft-client-0, createdTimeMs=1690290424955, requestBuilder=VoteRequestData(clusterId='MkU3OEVBNTcwNTJENDM2Qg', topics=[TopicData(topicName='__cluster_metadata', partitions=[PartitionData(partitionIndex=0, candidateEpoch=12, candidateId=0, lastOffsetEpoch=0, lastOffset=0)])])) (kafka.raft.RaftSendThread)
[2023-07-25 13:07:05,435] ERROR Request OutboundRequest(correlationId=518, data=VoteRequestData(clusterId='MkU3OEVBNTcwNTJENDM2Qg', topics=[TopicData(topicName='__cluster_metadata', partitions=[PartitionData(partitionIndex=0, candidateEpoch=12, candidateId=0, lastOffsetEpoch=0, lastOffset=0)])]), createdTimeMs=1690290424955, destinationId=2) failed due to authentication error (kafka.raft.KafkaNetworkChannel)
org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256
[2023-07-25 13:07:05,435] ERROR [RaftManager nodeId=0] Unexpected error NETWORK_EXCEPTION in VOTE response: InboundResponse(correlationId=518, data=VoteResponseData(errorCode=13, topics=[]), sourceId=2) (org.apache.kafka.raft.KafkaRaftClient)
[2023-07-25 13:07:05,471] INFO [SocketServer listenerType=CONTROLLER, nodeId=0] Failed authentication with /10.244.5.196 (channelId=10.244.5.194:9093-10.244.5.196:41032-10) (Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256) (org.apache.kafka.common.network.Selector)