Error while publishing messages after adding new brokers

Ops
1

Hello,
Currently we have a 3 node kafka cluster running on kubernetes and now we are planning to increase brokers to 5 due to increasing topics.
After adding two more brokers I am getting below error when I try to publish message on partition that is hosted on new brokers.

There are no errors in brokers logs or zookeeper logs. Do we need to do any additional setup for adding new brokers to existing cluster?

[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 4 (broker-4-kafka/IP:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.
2

You’ll need to share your k8s configs.

As the error says, are you using custom TLS certs that you’ve not distributed to the new brokers?

4

Thanks for the reply.

Finally we were able to find out the issue. There was some issue with kubernetes DNS service and DNS records for new brokers are not created.
After restarting the DNS service records got created and now all brokers are working.