Configuring connector from Confluent Hub with Confluent Cloud

Confluent Cloud
1

hello, so as the title goes, I am trying to figure out how to configure a connector, downloaded from Confluent Hub, to work with a cluster running in Confluent Cloud?

I’ve had a look at few articles on Confluent & almost all of them are either using Docker or Confluent Platform as example.

My search skills may not be giving me good results hence I am asking for help/suggestions, if someone can point out few resources showing how to get a connector from Confluent Cloud working with Confluent Cloud.

How do I figure out what settings need to be configured for a given connector, do I have read the JAR file?

2

@rmoff has a blog post, that you may have already seen, which shows how to hook up a self-managed connector to Confluent Cloud. He does use Docker, but the configuration properties should be the same. As for properties specific to a certain connector, I would suggest checking with the connector’s author. If you would like to share the connector you are working with, I can try to help find more info.

Thanks,
Dave

1 Like
3

thanks Dave, yes I read that blog by Robin, it is well written & clears up few doubts that I had. I understand that a worker node will be required for connectors but what I also noticed is that there are 2 connector types. Supported by Confluent Cloud to run locally on managed Kafka cluster & the others for which worker nodes are required (please do correct me if I am wrong). I was hoping that Confluent Cloud had a working example for both cases that people could read & follow. Robin’s blog is no doubt a good start so thanks @rmoff

This is one of three connectors that I am trying to find properties for –

The other one is –

For that matter, how can I find properties for connectors if it is not in the Confluent hub download page. Or not clearly written in ReadMe file.

4

I think you might be misunderstanding the two types of connectors, in relation to Confluent Cloud, or maybe I’m misunderstanding you, but in any event…

With Confluent Cloud you can use fully-managed connectors, which run in Confluent Cloud, and self-managed connectors which run on a connect worker that you are hosting, either locally, or in a self-managed cloud environment (eg. on an EC2 instance).

The two connectors you are referring to are both self-managed, at least for now. So you will need to run a Connect cluster and configure workers for these connectors. The two easiest ways to do this are Docker, which Robin shows, or Confluent Platform. But however you do it, you do need to have Kafka Connect running somewhere and that’s where you’d configure your workers.

The S3 sink connector is fairly well documented at: https://docs.confluent.io/kafka-connect-s3-sink/current/index.html?ajs_aid=69cc65ab-86dc-4105-9fb9-c67133f77f81&ajs_uid=162281

There is a section on that page that links to the full set of config settings, but here’s the link to save some time: https://docs.confluent.io/kafka-connect-s3-sink/current/configuration_options.html

The settings and other information on using the New Relic connector can be found on github. It’s a bit tricky since it’s not in the main readme for the repo, but down a level. Here’s the link:

Let me know if that clears things up at all.

Dave

1 Like
5

thanks @daveklein for your response, you got my question right, I was wondering about managed vs self-managed connectors in use with Confluent Cloud & your reply cleared things up for me :slightly_smiling_face:
Sorry, if I didn’t phrase the question properly :slightly_frowning_face:
The connector config part of my question was from observability perspective, to understand how a certain connector is performing & what metrics are available for monitoring? Thanks for pointing out the section to read about those configurations.