Currently we have a 3 node kafka cluster running on kubernetes and now we are planning to increase brokers to 5 due to increasing topics.
After adding two more brokers I am getting below error when I try to publish message on partition that is hosted on new brokers.
There are no errors in brokers logs or zookeeper logs. Do we need to do any additional setup for adding new brokers to existing cluster?
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 4 (broker-4-kafka/IP:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.