Cant figure out why I am getting advertised.listeners error

Architecture and Design
1

I am getting the following error but do not understand why. Anyone have any ideas?

[2024-04-22 13:16:37,578] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
java.lang.IllegalArgumentException: requirement failed: The advertised.listeners config must not contain KRaft controller listeners from controller.listener.names when process.roles contains the broker role because Kafka clients that send requests via advertised listeners do not send requests to KRaft controllers – they only send requests to KRaft brokers.
at scala.Predef$.require(Predef.scala:337)
at kafka.server.KafkaConfig.validateAdvertisedListenersDoesNotContainControllerListenersForKRaftBroker$1(KafkaConfig.scala:2352)
at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:2420)
at kafka.server.KafkaConfig.(KafkaConfig.scala:2290)
at kafka.server.KafkaConfig.(KafkaConfig.scala:1639)
at kafka.Kafka$.buildServer(Kafka.scala:71)
at kafka.Kafka$.main(Kafka.scala:90)
at kafka.Kafka.main(Kafka.scala)

The server properties file looks like this.

process.roles=broker,controller

node.id=1

controller.quorum.voters=0@localhost:19092,1@localhost:19093,2@localhost:19094

ssl.keystore.location=/opt/kafka/kafka-poc.mfs.com/kafka-poc.mfs.com.jks
ssl.keystore.password=KeepMeSecure
ssl.key.password=KeepMeSecure
ssl.truststore.location=/opt/kafka/kafka-poc.mfs.com/kafka-poc.mfs.com.p12
ssl.truststore.password=KeepMeSecure
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.client.auth=required

listeners=SSL://:9092,CONTROLLER://:19092
advertised.listeners=SSL://bosukafkbrkrd01:9092,CONTROLLER://bosukafkbrkrd01:19092
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

controller.listener.names=CONTROLLER

sasl.enabled.mechanisms=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN
security.inter.broker.protocol=SASL_SSL
authorizer.class.name=kafka.security.authorizer.AclAuthorizer

2

You just need to remove CONTROLLER://bosukafkbrkrd01:19092 from the advertised.listeners value, but leave the controller listener in the listeners value.

From the KRaft socket server settings documentation on listeners:

For controllers in combined mode (i.e., process.roles=broker,controller), you should list the controller listeners as well as the broker listeners

And below that on controller.listener.names:

For KRaft controllers in isolated or combined mode, the node will listen as a KRaft controller on all listeners that are listed for this property, and each must appear in the listeners property. They shouldn’t appear in the advertised.listeners property, which is used in ZooKeeper mode.

3

I have the config set as such. Still getting the error. The advertised listener is just the broker.

Thats what s weird.

controller.quorum.voters=1@bosukafkbrkrd01:19092,2@bosukafkbrkrd01:19093,3@bosukafkbrkrd01:19094
listeners=BROKER://:9092,CONTROLLER://:19092
advertised.listeners=BROKER://:9092
inter.broker.listener.name=BROKER
controller.listener.names=CONTROLLER

4

You get the same exact error or something different? I wouldn’t expect you to hit The advertised.listeners config must not contain KRaft controller listeners from controller.listener.names given the latest snippet.

One other issue I see in the initial snippet is that node.id=1 for the controller listening on port 19092 but controller.quorum.voters has 0@localhost:19092,1@localhost:19093

5

You do?

I did a grep of the three server property files for the brokers and i see.

The errors I previously sent are the same.

kafka@bosukafkbrkrd01: /opt/kafka/kafka_2.13-3.7.0/config/kraft$ grep voter server*.properties
server1.properties:controller.quorum.voters=1@bosukafkbrkrd01:19092,2@bosukafkbrkrd01:19093,3@bosukafkbrkrd01:19094
server2.properties:controller.quorum.voters=1@bosukafkbrkrd01:19092,2@bosukafkbrkrd01:19093,3@bosukafkbrkrd01:19094
server3.properties:controller.quorum.voters=1@bosukafkbrkrd01:19092,2@bosukafkbrkrd01:19093,3@bosukafkbrkrd01:19094

6

You’ll want to make sure that the node.id and controller port in the listeners properties match what’s in controller.quorum.voters. This wasn’t the case in the initial config snippet shared, and it’s not clear based on the latest grep output if this has been corrected

Aside from that, I’m not sure what might be the issue. If you can share a more complete repro setup (the 3 full config files and any other repro specifics), I can take a closer look.

7

Hmmm … below are the three server prop files involved.

Any help you can provide would be greatly appreciated.

server1.properties
process.roles=broker,controller
node.id=1
controller.quorum.voters=1@bosukafkbrkrd01:19092,2@bosukafkbrkrd01:19093,3@bosukafkbrkrd01:19094
listeners=BROKER://:9092,CONTROLLER://:19092
advertised.listeners=BROKER://:9092
inter.broker.listener.name=BROKER
controller.listener.names=CONTROLLER
listener.security.protocol.map=BROKER:SASL_SSL,CONTROLLER:SASL_SSL
listener.name.controller.ssl.client.auth=required
listener.name.broker.ssl.client.auth=required
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
ssl.keystore.location=/opt/kafka/kafka-poc.mfs.com/kafka-poc.mfs.com.jks
ssl.keystore.password=KeepMeSecure
ssl.key.password=KeepMeSecure
ssl.truststore.location=/opt/kafka/kafka-poc.mfs.com/kafka-poc.mfs.com.p12
ssl.truststore.password=KeepMeSecure
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.client.auth=required

authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
super.users=User:admin
sasl.enabled.mechanisms=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.mechanism.controller.protocol=PLAIN
listener.name.controller.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required
username=“admin”
password=“secret000”
user_admin=“secret000”;
listener.name.broker.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required
username=“admin”
password=“secret000”
user_admin=“secret000”;
log.dirs=/opt/kafka/kraft-combined-logs-1
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

server2.properties
process.roles=broker,controller
node.id=2
controller.quorum.voters=1@bosukafkbrkrd01:19092,2@bosukafkbrkrd01:19093,3@bosukafkbrkrd01:19094
listeners=BROKER://:9093,CONTROLLER://:19093
advertised.listeners=BROKER://:9092
inter.broker.listener.name=BROKER
controller.listener.names=CONTROLLER
listener.security.protocol.map=BROKER:SASL_SSL,CONTROLLER:SASL_SSL
listener.name.controller.ssl.client.auth=required
listener.name.broker.ssl.client.auth=required
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
ssl.keystore.location=/opt/kafka/kafka-poc.mfs.com/kafka-poc.mfs.com.jks
ssl.keystore.password=KeepMeSecure
ssl.key.password=KeepMeSecure
ssl.truststore.location=/opt/kafka/kafka-poc.mfs.com/kafka-poc.mfs.com.p12
ssl.truststore.password=KeepMeSecure
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.client.auth=required

authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
super.users=User:admin
sasl.enabled.mechanisms=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.mechanism.controller.protocol=PLAIN
listener.name.controller.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required
username=“admin”
password=“secret000”
user_admin=“secret000”;
listener.name.broker.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required
username=“admin”
password=“secret000”
user_admin=“secret000”;
log.dirs=/opt/kafka/kraft-combined-logs-2
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

server3.properties
process.roles=broker,controller
node.id=3
controller.quorum.voters=1@bosukafkbrkrd01:19092,2@bosukafkbrkrd01:19093,3@bosukafkbrkrd01:19094
listeners=BROKER://:9094,CONTROLLER://:19094
advertised.listeners=BROKER://:9092
inter.broker.listener.name=BROKER
controller.listener.names=CONTROLLER
listener.security.protocol.map=BROKER:SASL_SSL,CONTROLLER:SASL_SSL
listener.name.controller.ssl.client.auth=required
listener.name.broker.ssl.client.auth=required
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
ssl.keystore.location=/opt/kafka/kafka-poc.mfs.com/kafka-poc.mfs.com.jks
ssl.keystore.password=KeepMeSecure
ssl.key.password=KeepMeSecure
ssl.truststore.location=/opt/kafka/kafka-poc.mfs.com/kafka-poc.mfs.com.p12
ssl.truststore.password=KeepMeSecure
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.client.auth=required

authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
super.users=User:admin
sasl.enabled.mechanisms=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.mechanism.controller.protocol=PLAIN
listener.name.controller.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required
username=“admin”
password=“secret000”
user_admin=“secret000”;
listener.name.broker.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required
username=“admin”
password=“secret000”
user_admin=“secret000”;
log.dirs=/opt/kafka/kraft-combined-logs-3
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

8

The only issue I see with these is that the advertised.listeners ports for nodes 2 and 3 should be corrected to 9093 and 9094, respectively. Fixing that should get you further along. I was able to start 3 servers without error on my laptop with close to these configs (just no security, and using localhost).

Also double checking that you formatted storage as in the quickstart here, i.e.:

kafka-storage.sh format -t  <GUID>  -c /path/to/server.properties

using the same GUID for all 3 properties files.

9

That did it . Things are up!

Can’t get topics list now tho.

kafka@bosukafkbrkrd01: /opt/kafka/kafka_2.13-3.7.0$ kafka-topics.sh --bootstrap-server bosukafkbrkrd01:9092 --list
Error while executing topic command : Timed out waiting for a node assignment. Call: listTopics
[2024-04-29 12:01:04,689] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics
(org.apache.kafka.tools.TopicCommand)

Logs show:
[2024-04-29 12:02:04,671] INFO [SocketServer listenerType=BROKER, nodeId=2] Failed authentication with /168.66.123.25 (channelId=168.66.122.158:9093-168.66.123.25:27386-60) (SSL handshake failed) (org.apache.kafka.common.network.Selector)

10

Figured that out … D’oh… Needed to define the adminclient-configs.conf and client-ssl.properties.

With those in place, I can get the results with the --command-config adminclient-configs.conf

Any way to make this a default behavior so I dont always have to type it in?

11

Glad you figured it out!

Not that I know of… Could you please share the command that required it? The reason I ask is that the help string says that this property “is used only with --bootstrap-server option for describing and altering broker configs,” but it sounds like you needed it to list topics too? Could be a doc bug if so.

12

For example:
kafka-acls.sh --bootstrap-server bosukafkbrkrd01:9092 --command-config /opt/kafka/kafka_2.13-3.7.0/config/kraft/adminclient-configs.conf --list

Since I would be doing this for all command, it would be nice to just accept the --command-config location as a default.

If I were to set an env variable or set the value in a config, that would be great.