We are trying to connect to our kafka cluster through our corporate proxy server.
(on prem client → proxy server → kafka cluster in cloud).
When testing the connection using CLI, there is no means to specify a proxy server to facilitate the connection.
Looking for assistance on how to configure the connection.
Thanks
Patrick
./bin/kafka-console-producer.sh --topic --producer.config ./config/producer.properties --bootstrap-server my.instance.name:xxxx
Hi @pgfitzpatrick,
The console producer doesn’t support proxying, and more generally the Kafka protocol isn’t conducive to using a proxy because clients need direct individual broker connectivity.
Dave
Hi Dave,
thanks for the prompt reply. Just to clarify - for some of our internal clients that have custom java consumers/producers; then the use of a proxy is not possible (direct conn is required)?
Would the same apply for confluent replicator?
replicator → Proxy Server - > Kafka Cluster
Thanks
Patrick
Hi @pgfitzpatrick,
thanks for the prompt reply. Just to clarify - for some of our internal clients that have custom java consumers/producers; then the use of a proxy is not possible (direct conn is required)?
Yes I’d say it’s not recommended. It might technically be possible with some proxy config and DNS creativity, but this wouldn’t be recommended. Quoting this SO answer from @OneCricketeer:
Any Proxy would need to know each individual address, and thus defeats the purpose of having a single load-balanced proxy anyway. The Kafka protocol handles its own load balancing and bootstrapping.
Would the same apply for confluent replicator?
replicator → Proxy Server - > Kafka Cluster
Yes, the same applies since Replicator is built on Connect which is built on the same Java Producer/Consumer client that your internal clients are using.
Dave
Connecting to a Kafka cluser through a corporate proxy server can be a bit tricky, but it’s doable. You might not have a direct option in the CLI, but you can work around it.
Consider setting up a proxy site that sits between your on-prem client and the Kafka cluster in the cloud. Then, configure your client to connect to the site instead. The site should handle the communication with the Kafka cluster, forwarding your requests.
Make sure to consult your corporate IT or network team for guidance on proxy server configurations and any security considerations. With the right setup, you should be able to establish a connection smoothly.