Considering most enterprises have IP exhaustion of their RFC 1918 from poor allocation practices over many year, its INSANE that Confluent Cloud requires a /16 to do a VNET deployment. What customer will need to spin up 65k nodes in a Kafka cluster???
If your trying to do Kafka Connect with Confluent Cloud and need to reach back to an on-premises databases there are no good options. Either gobble up a /16 of valued Private IP space or you have to punch a hole in your firewall to let traffic through (really???). Not to mention that renders ExpressRoute useless.
What about conecting to Service Endpoints / Private Link to Storage accounts or databases in Azure? Without a VNET you are stuck opening the firewall on them. Private Link to Confluent Cloud is just about giving a unidirectional private IP towards cloud but does NOTHING for connecting to external things that aren’t on the internet.
The other option of running Confluent Platform in your on premise data center and paying its exorbinant 3x licensing costs compared to Confluent Cloud along with having to run a bunch of hardware and storage isn’t really a great option either.
Is there any way to lower that VNET size manually?
Hello @dalehypereon and welcome to the Confluent Community Forum!
Currently, the /16 CIDR is a hard requirement to set up VNet peeting on Azure. This may change in the future though.
For connecting (via Kafka Connect) to Private Link endpoints for Azure storage / databases, there’s a hybrid solution where you run the connector in your VNet. Many Confluent Cloud customers take this approach. Note that there isn’t a license cost for Confluent Commercial Connectors with this setup. From here:
Self-managed connectors have no billing mechanism themselves. However, note that using self-managed connectors may incur ingress, egress, and storage charges for your Kafka clusters running in Confluent Cloud.
We have Confluent Cloud on Azure with PrivateLink setup as well. We’re planning for additional Connectors, some of which have to access data inside inside Azure private VNETs or on-premise data centers.
-
Some data sources are on-premise. Even when (if?) Connectors can access Azure resources over PrivateLink, that still doesn’t help with on-premises data sources.
-
Unable to whitelist IPs used by Connectors in networking rules. For accessing Azure resources, we could enable public access and whitelist IPs used by Connectors. Unfortunately, clusters with Private Networking don’t support static egress IPs.
Are there features on the Roadmap to improve security of ConfCloud → Azure/OnPremise networking?