I realized that username=“kafka” password=“kafka-password”
used for communication between brokers.
But for what purpose user_kafka=“kafka-password” line?
Based on the Apache Kafka documentation here, they’re both used for communication between brokers. The username / password properties are used to initiate an inter-broker connection (say on Broker A that is connecting to Broker B), and the corresponding user_<username value> property is used to validate the connection request (on Broker B).
The properties username and password in the KafkaServer section are used by the broker to initiate connections to other brokers. In this example, admin is the user for inter-broker communication. The set of properties user_userName defines the passwords for all users that connect to the broker and the broker validates all client connections including those from other brokers using these properties.
I’d venture a guess that there are separate initiation / validation properties in order to support zero-downtime rolling password changes. E.g., say you wanted to change the inter-broker credentials to username kafka2 with password kafka-password-2, then you could:
add user_kafka2 property to all broker configs and rolling restart them
change username to kafka2 and password to kafka-password-2 on all brokers and rolling restart them
remove the user_kafka property on all brokers and rolling restart them
In the middle of step 2 you’d have some brokers initiating connections with username kafka and some initiating with kafka2 but everyone can connect given what happens in step 1. Then step 3 is just cleanup.