Curious whether there are any caveats to connecting a Kstreams client to Kafka through a VPN gateway and/or an API gateway.
Can I bootstrap a connector to Kafka as long as I can connect to Zookeeper and the Kafka leader? This use case I’m thinking is an on-prem client <-> Azure cluster. The Azure cluster could be CC and/or AKS and/or VM, with a network route through ExpressRoute and/or VPN tunnel.
I think this design is doable as long that you ensure bi-directional communication between the client and the Kafka cluster, as well as if you ensure that all this traffic goes over TCP. This might mean that you need to set up security groups to allow both ingress and egress routes. I don’t have much experience doing this with Azure but on AWS there would be some complications related to where the endpoint should be terminated that has to be handled. Not sure if that translates to how Azure does things as well.
But if you have the option to do things differently, I would recommend change the design and have your clients on-prem talking to a local Kafka cluster and then having this local Kafka cluster replicated to your cloud Kafka cluster — either using MirrorMaker 2.0 or Confluent Replicator.
It’s less about a design and more about constraints from my org. Given my druthers, I’d use an MRC but there are other forces at play. One of those forces might be to force an integration across network domains and I’d like to rule out an attempt to do what I described, if the terminating endpoints break the ack behaviour of a streams client.
Thx.