Hello,
There is one scenario that is not achieveable in kafka.
- There is a setup of three kafka servers in cluster. Kafka Servers has private IPs and is only accessible from its own VPN say KAFKA VPN.
- There is one public IP that is accessible when machine is connected to KAFKA VPN + Office Netowork. This public IP is only used to NAT the private IPs so that over the internet people can access it from Office Network.
- There is one consumer at Office Network. Now that consumer can consume topics from Kafka using public IP when it is connected to KAFKA VPN.
Now I want to connect Public IP by disconnecting KAFKA VPN. Is that possible ? Network is working fine. I have checked Trace Route and Reversed Trace Route.
I think there is some problem with producer / kafka server / consumer configuration
NAT Configuration::
10.XX.XX.XX:9092 -> AA.XX.XX.XX:9095
10.XX.XX.XY:9092 -> AA.XX.XX.XX:9093
10.XX.XX.XZ:9092 -> AA.XX.XX.XX:9094
Producer Configuration::
bootstrap-servers=10.XX.XX.XX:9092,10.XX.XX.XY:9092,10.XX.XX.XZ:9092
Kafka Configuration::
listeners=SASL_SSL://0.0.0.0:9092,EXTERNAL://0.0.0.0:9095
advertised.listeners=SASL_SSL://10.XX.XX.XX:9092,EXTERNAL://AA.XX.XX.XX:9095
listener.security.protocol.map=SASL_SSL:SASL_SSL,EXTERNAL:SASL_SSL
Consumer Configuration::
sh kafka-console-consumer.sh --bootstrap-server AA.XX.XX.XX:9093,AA.XX.XX.XX:9094,AA.XX.XX.XX:9095 --topic test --consumer.config consumer.properties
Error that I am getting when I am not connected to KAFKA VPN::
[2022-03-02 17:35:58,236] WARN [Consumer clientId=consumer-test_group-1, groupId=test_group] Connection to node 2147483646 (10.XX.XX.XX/10.XX.XX.XX:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-03-02 17:36:26,421] WARN [Consumer clientId=consumer-test_group-1, groupId=test_group] Connection to node 2 (10.XX.XX.XY/10.XX.XX.XY:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-03-02 17:36:47,467] WARN [Consumer clientId=consumer-test_group-1, groupId=test_group] Connection to node 3 (10.XX.XX.XZ/10.XX.XX.XZ:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient
Now some of the questionaries are ::
- Even if I am using Public IP to connect to Kafka, Why it is showing Private IP in logs ?
- Should bootstrap servers in producer and bootstrap servers in consumer must be exactly same ?
- How can I check the response of metadata request. Does kafka have any program to check ? Like it has for kafka-console-consumer.
- How should I fix this ?
- Consider SASL and SSL properties and certificate in place. Consider all the network are in its place.
** I have attached architecture details in this forum
It would be great if someone help me on this
Thanks,
Milan K