Google Sinks workaround in AWS Kafka Cluster

Kafka Connect Managed Connectors
1

Hello,

I’m having some issues trying to send data from a Kafka cluster, configured with an AWS provider, into a Google Storage bucket. I tried installing the Google Storage sink connector locally and executing it.

The error is the following:
Error: unable to validate plugin type: connect service: connector plugin GcsSink unavailable on requested cloud provider aws

Any idea on how to sort this out?

Thank you in advance.

2

Howdy there,

I will assume that you’re talking about Confluent Cloud and its managed Google Cloud Storage sink connector. If this is the case, you need to create your Kafka cluster with Google Cloud as the provider, as you can read in the product documentation:

image
image2212×734 124 KB

:arrow_right: “A GCP GCS bucket in the same region as your Confluent Cloud cluster.”

— @riferrei

3

Hi Ricardo,

Indeed, you are right. I’m trying to use as much as possible Confluent Cloud but I might need to shift to Confluent CLI to overcome this one.

The deal, and not negotiable, is that I need my cluster with the AWS provider to send the information to GCS Bucket.

I followed this TUTORIAL that helped me a lot, nonetheless, I’m curious if it’s possible to proceed using Confluent Cloud.

4

Fair enough. This will become a trade-off between using the managed connector and hosting your own. The managed connector has its own sets of validations before pushing data to the destination — hence why it is not working. If you want to keep on AWS, you will need to host your own connector instance somewhere, license it properly, and maintain this deployment.

You may reach a point to say that it would be simpler to move the cluster to Google Cloud :sweat_smile:

5

I do understand it all, and thank you very much for your availability to help.

What I can’t change at the moment is that there’s another part of the company that’s using AWS so, if I move the Kafka cluster to GCP, they’ll be the ones with the need to manage all of this.

I wonder if GCP doesn’t have a component, like pub/sub, to easily configure and read information from the Kafka producer without. That way, it’s would not be required to create and manage a self-managed connector.