Temporarily enable Plaintext on SSL Config

Rand · 6 February 2025 11:59

Hi,

sorry if this should be obvious but I can’t seem to find this either

The SSL enabled config is working but I need to debug it.
I now need to debug some things and want to connect to the kafka cluster via cli.
Since I cant find how to use SSL from the cli I temporarily want to enable a Plaintext listener for the controller to connect to:

-e KAFKA_NODE_ID=1 \
-e KAFKA_PROCESS_ROLES=‘controller’ \
-e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=‘CONTROLLER:SSL, SSLINTERNAL:SSL, PLAINTEXT:PLAINTEXT’ \
-e KAFKA_CONTROLLER_QUORUM_VOTERS=‘1@host1:9092,2@host2:9092,3@host3:9092’ \
-e KAFKA_INTER_BROKER_LISTENER_NAME=‘SSLINTERNAL’ \
-e KAFKA_CONTROLLER_LISTENER_NAMES=‘CONTROLLER, PLAINTEXT’ \
-e KAFKA_LISTENERS=‘CONTROLLER://:9092, PLAINTEXT://:9094’ \
-e CLUSTER_ID=‘clusterid’ \

Unfortunatly this does not work with

Exception in thread “main” java.lang.IllegalArgumentException: requirement failed: The listeners config must only contain KRaft controller listeners from controller.listener.names when process.roles=controller at scala.Predef$.require(Predef.scala:337) at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:1341) at kafka.server.KafkaConfig.(KafkaConfig.scala:1223) at kafka.server.KafkaConfig.(KafkaConfig.scala:545) at kafka.tools.StorageTool$.$anonfun$execute$1(StorageTool.scala:72) at scala.Option.flatMap(Option.scala:283) at kafka.tools.StorageTool$.execute(StorageTool.scala:72) at kafka.tools.StorageTool$.main(StorageTool.scala:53) at kafka.tools.StorageTool.main(StorageTool.scala)

How can I add a plaintext listener to a KRaft enabled controller?

Thanks

dtroiano · 6 February 2025 17:54

I don’t follow what the “cli” aspect of this. Are you trying to configure it so that brokers communicate with the controller over plaintext?

What Docker image and version are you using? I tried this Docker Compose to see if controller startup would fail with the same error but it starts up without any error:

---
services:
  controller-1:
    image: confluentinc/cp-server:latest
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: 'controller'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:9093'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER,PLAIN'
      KAFKA_LISTENERS: 'CONTROLLER://controller-1:9093,PLAIN://controller-1:9094'
      CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'

Rand · 7 February 2025 06:48

H Dave,

I want to connect to Kafka using the provided binaries in the docker images to debug my issue.
./kafka-replica-verification, kafka-topics and maybe some of the perf tests

When trying to run those on vontroller to controller port or on broker to broker port I get errors

Broker error

sh-4.4$ ./kafka-replica-verification --broker-list localhost:9093
[2025-02-07 06:21:05,048] ERROR Uncaught exception in thread ‘kafka-admin-client-thread | adminclient-1’: (org.apache.kafka.common.utils.KafkaThread)
java.lang.OutOfMemoryError: Java heap space
at java.base/java.nio.HeapByteBuffer.(Unknown Source)
at java.base/java.nio.ByteBuffer.allocate(Unknown Source)
at org.apache.kafka.common.memory.MemoryPool$1.tryAllocate(MemoryPool.java:30)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:102)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:462)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:412)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:678)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:580)
at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:595)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1524)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1455)
at java.base/java.lang.Thread.run(Unknown Source)
org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited. Call: listTopics

controller error

sh-4.4$ ./kafka-replica-verification --broker-list localhost:9092
[2025-02-07 06:21:32,325] ERROR Uncaught exception in thread ‘kafka-admin-client-thread | adminclient-1’: (org.apache.kafka.common.utils.KafkaThread)
java.lang.OutOfMemoryError: Java heap space
at java.base/java.nio.HeapByteBuffer.(Unknown Source)
at java.base/java.nio.ByteBuffer.allocate(Unknown Source)
at org.apache.kafka.common.memory.MemoryPool$1.tryAllocate(MemoryPool.java:30)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:102)

I’ve read that the memory errors are misleading and that it means that it cannot connect, quite like due to the requirement of SSL which i dont have on the cli.
I could no either try to config the cli to use ssl and use the listener I have (but no idea how to do that since I dont see any options to pass in certs et al, and I dont have a config file to use either);
or I can set up a temporary Plaintext listener on the controller (which I think is what I need to connect to usually?) which I then would use to use kafka-topics & col. The latter is what we did with our ZK based cluster but i can’t seem to get it going with the KRaft based one.

With your example code I get

No security protocol defined errro

Exception in thread “main” java.lang.IllegalArgumentException: Error creating broker listeners from ‘CONTROLLER://:9092,PLAIN://:9094’: No security protocol defined for listener PLAIN at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:216) at kafka.server.KafkaConfig.listeners(KafkaConfig.scala:1104) at kafka.server.KafkaConfig.(KafkaConfig.scala:754) at kafka.server.KafkaConfig.(KafkaConfig.scala:545) at kafka.tools.StorageTool$.$anonfun$execute$1(StorageTool.scala:72) at scala.Option.flatMap(Option.scala:283) at kafka.tools.StorageTool$.execute(StorageTool.scala:72) at kafka.tools.StorageTool$.main(StorageTool.scala:53) at kafka.tools.StorageTool.main(StorageTool.scala) Caused by: java.lang.IllegalArgumentException: No security pro

but when adding that it works.

-e KAFKA_PROCESS_ROLES='controller' \
-e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP='CONTROLLER:SSL, SSLINTERNAL:SSL, PLAIN:PLAINTEXT' \
-e KAFKA_CONTROLLER_QUORUM_VOTERS='1@host1:9092,2@host2:9092,3@host3:9092' \
-e KAFKA_INTER_BROKER_LISTENER_NAME='SSLINTERNAL' \
-e KAFKA_CONTROLLER_LISTENER_NAMES='CONTROLLER,PLAIN' \
-e KAFKA_LISTENERS='CONTROLLER://:9092,PLAIN://:9094' \

Thanks
O/c I run into the next errors right away but thats a different topic

system · 14 February 2025 06:49

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
SSL Woes II (repost) Ops	13	3157	8 January 2025
Kafka KRAFT SASL_PLAINTEXT Containers	3	7205	10 September 2024
Unable to customize the confluent operator for listener.security.protocol.map from PLAINTEXT to SSL Confluent Cloud	0	1167	25 October 2023
Kraft with SASL/PLAINTEXT doesn't work with SCRAM-SHA-256 Containers	0	399	28 August 2024
SASL_PLAIN for client connection configuration problems Java Clients	2	6880	4 December 2021

Temporarily enable Plaintext on SSL Config

Related topics