Kraft with SASL/PLAINTEXT doesn't work with SCRAM-SHA-256

Sakib_Farhad · 28 August 2024 06:27

We are trying to setup a 3 node cluster with confluentinc/cp-kafka:7.7.0. Following are the docker-compoes and jaas files.

services:
  kafka-1:
    user: 0:0
    image: confluentinc/cp-kafka:7.7.0
    hostname: kafka-1
    container_name: kafka-1
    ports:
      - 29092:9092
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: "broker,controller"
      KAFKA_LISTENERS: BROKER://kafka-1:9092,CONTROLLER://kafka-1:9093
      KAFKA_ADVERTISED_LISTENERS: BROKER://kafka-1:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: BROKER:SASL_PLAINTEXT,CONTROLLER:SASL_PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093
      CLUSTER_ID: "4L6g3nShT-eMCtK--X86sw"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_LOG_DIRS: "/tmp/kraft-combined-logs"
      KAFKA_SASL_ENABLED_MECHANISMS: SCRAM-SHA-256,PLAIN
      KAFKA_SASL_MECHANISM_CONTROLLER_PROTOCOL: PLAIN
      KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: SCRAM-SHA-256
      KAFKA_INTER_BROKER_LISTENER_NAME: BROKER
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_AUTHORIZER_CLASS_NAME: org.apache.kafka.metadata.authorizer.StandardAuthorizer
      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "false"
      KAFKA_SUPER_USERS: User:admin
      KAFKA_OPTS: "-Djava.security.auth.login.config=/etc/kafka/secrets/jaas.conf"
    volumes:
      - ./kafka/config/jaas.conf:/etc/kafka/secrets/jaas.conf
      - ./kafka/config/admin.config:/etc/kafka/admin.config

  kafka-2:
    user: 0:0
    image: confluentinc/cp-kafka:7.7.0
    hostname: kafka-2
    container_name: kafka-2
    ports:
      - 39092:9092
    environment:
      KAFKA_NODE_ID: 2
      KAFKA_PROCESS_ROLES: "broker,controller"
      KAFKA_LISTENERS: BROKER://kafka-2:9092,CONTROLLER://kafka-2:9093
      KAFKA_ADVERTISED_LISTENERS: BROKER://kafka-2:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: BROKER:SASL_PLAINTEXT,CONTROLLER:SASL_PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093
      CLUSTER_ID: "4L6g3nShT-eMCtK--X86sw"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_LOG_DIRS: "/tmp/kraft-combined-logs"
      KAFKA_SASL_ENABLED_MECHANISMS: SCRAM-SHA-256,PLAIN
      KAFKA_SASL_MECHANISM_CONTROLLER_PROTOCOL: PLAIN
      KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: SCRAM-SHA-256
      KAFKA_INTER_BROKER_LISTENER_NAME: BROKER
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_AUTHORIZER_CLASS_NAME: org.apache.kafka.metadata.authorizer.StandardAuthorizer
      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "false"
      KAFKA_SUPER_USERS: User:admin
      KAFKA_OPTS: "-Djava.security.auth.login.config=/etc/kafka/secrets/jaas.conf"
    volumes:
      - ./kafka/config/jaas.conf:/etc/kafka/secrets/jaas.conf
      - ./kafka/config/admin.config:/etc/kafka/admin.config

  kafka-3:
    user: 0:0
    image: confluentinc/cp-kafka:7.7.0
    hostname: kafka-3
    container_name: kafka-3
    ports:
      - 49092:9092
    environment:
      KAFKA_NODE_ID: 3
      KAFKA_PROCESS_ROLES: "broker,controller"
      KAFKA_LISTENERS: BROKER://kafka-3:9092,CONTROLLER://kafka-3:9093
      KAFKA_ADVERTISED_LISTENERS: BROKER://kafka-3:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: BROKER:SASL_PLAINTEXT,CONTROLLER:SASL_PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093
      CLUSTER_ID: "4L6g3nShT-eMCtK--X86sw"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_LOG_DIRS: "/tmp/kraft-combined-logs"
      KAFKA_SASL_ENABLED_MECHANISMS: SCRAM-SHA-256,PLAIN
      KAFKA_SASL_MECHANISM_CONTROLLER_PROTOCOL: PLAIN
      KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: SCRAM-SHA-256
      KAFKA_INTER_BROKER_LISTENER_NAME: BROKER
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_AUTHORIZER_CLASS_NAME: org.apache.kafka.metadata.authorizer.StandardAuthorizer
      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "false"
      KAFKA_SUPER_USERS: User:admin
      KAFKA_OPTS: "-Djava.security.auth.login.config=/etc/kafka/secrets/jaas.conf"
    volumes:
      - ./kafka/config/jaas.conf:/etc/kafka/secrets/jaas.conf
      - ./kafka/config/admin.config:/etc/kafka/admin.config

jaas.conf

KafkaServer {
    org.apache.kafka.common.security.scram.ScramLoginModule required
    username="admin"
    password="admin"
    user_admin="admin"
    user_usera="usera"
    user_userb="userb";
};
KafkaClient {
    org.apache.kafka.common.security.scram.ScramLoginModule required
    username="admin"
    password="admin"
    user_admin="admin";
};

admin.config

org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="admin";
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256

Here the issue we are facing is when we do the following

KAFKA_SASL_ENABLED_MECHANISMS: SCRAM-SHA-256,PLAIN
KAFKA_SASL_MECHANISM_CONTROLLER_PROTOCOL: PLAIN
KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: SCRAM-SHA-256

we get error such as

log 1

[2024-08-28 05:36:35,838] WARN [RaftManager id=1] Connection to node 2 (kafka-2/172.18.0.3:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. (org.apache.kafka.clients.NetworkClient)
[2024-08-28 05:36:35,839] INFO [RaftManager id=1] Node 3 disconnected. (org.apache.kafka.clients.NetworkClient)
[2024-08-28 05:36:35,839] WARN [RaftManager id=1] Connection to node 3 (kafka-3/172.18.0.2:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. (org.apache.kafka.clients.NetworkClient)
[2024-08-28 05:36:35,938] INFO [MetadataLoader id=1] initializeNewPublishers: the loader is still catching up because we still don’t know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader)
[2024-08-28 05:36:36,038] INFO [MetadataLoader id=1] initializeNewPublishers: the loader is still catching up because we still don’t know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader)
[2024-08-28 05:36:36,139] INFO [MetadataLoader id=1] initializeNewPublishers: the loader is still catching up because we still don’t know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader)
[2024-08-28 05:36:36,239] INFO [MetadataLoader id=1] initializeNewPublishers: the loader is still catching up because we still don’t know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader)
[2024-08-28 05:36:36,299] INFO [RaftManager id=1] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient)

and if we do the following

KAFKA_SASL_ENABLED_MECHANISMS: SCRAM-SHA-512,PLAIN
KAFKA_SASL_MECHANISM_CONTROLLER_PROTOCOL: SCRAM-SHA-512
KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: SCRAM-SHA-512

we get following

log 2

[2024-08-27 12:52:47,489] ERROR [RaftManager id=1] Connection to node 2 (kafka-2/172.19.0.4:9093) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512 (org.apache.kafka.clients.NetworkClient)
[2024-08-27 12:52:47,491] ERROR [kafka-1-raft-outbound-request-thread]: Failed to send the following request due to authentication error: ClientRequest(expectResponse=true, callback=org.apache.kafka.raft.KafkaNetworkChannel$$Lambda$672/0x0000757f303bed20@6d53e485, destination=2, correlationId=0, clientId=raft-client-1, createdTimeMs=1724763166889, requestBuilder=VoteRequestData(clusterId=‘4L6g3nShT-eMCtK–X86sw’, topics=[TopicData(topicName=‘__cluster_metadata’, partitions=[PartitionData(partitionIndex=0, candidateEpoch=1, candidateId=1, lastOffsetEpoch=0, lastOffset=0)])])) (org.apache.kafka.raft.KafkaNetworkChannel$SendThread)

Has anyone done this? Is this a bug?

Topic		Replies	Views
Kafka KRAFT SASL_PLAINTEXT Containers	3	6594	10 September 2024
Self-hosted Kafka with KRaft, SSL and SASL (scram-sha-256) Ops	7	7957	17 May 2024
Docker Compose using SASL_PLAINTEXT and SCRAM-SHA-256 Containers	3	11257	21 December 2021
Kafka brokers SCRAM-SHA-256 with kraft mode Ops	0	1362	4 February 2024
ClusterAuthorizationException with User:ANONYMOUS when setting up SASL with Kraft and ACL Containers	7	4257	24 October 2023

Kraft with SASL/PLAINTEXT doesn't work with SCRAM-SHA-256

Related topics