Slight confusion in running Kafka connect

I have a slight bit of confusion when i start to run Kafka connect on my own.

Here’s the scenario, I have a Kafka Cluster running in Confluent, I want to run Connect in my AWS EC2 machine and run AWS Sink connector for testing it in a standalone mode before starting in a Distributed mode.

I have read lot of documentation, after reading i thought i should use confluent-hub to download connector jar and then i can run.

But if i download the connector it gives me couple of warnings , i have searched the mentioned directories with confluent-hub, i dont get the idea on how to use confluent-hub properly for my use-case.

Then I thought its not correct way to do it, then i downloaded the Apache Kafka CLI setup and have seen that there are some options to run connect with standalone vs distributed mode.

But I dont get the idea on how do i get nice Connect worker running with this way, then POST some of the connector configuration using the REST API which is offered by connect.

But rather Its like running the process background , I don’t want to run it that way.(Capture — Postimages)

and i also seen confluent CLI is which looks bit different than confluent-hub which is again bit challenging for me to understand.
Can someone enlighten with simpler understanding. What’s the right way for me to run this and understand it before i move on start with Container solution.

Hi @srikanth597 . Welcome to the Confluent Community Forum!

Are you using Confluent Platform?

Yes I am using Confluent to run the Kafka, and Kafka Connect worker inside AWS EC2.

Equivalent Text for the Images.
First command :
confluent-hub install confluentinc/kafka-connect-aws-lambda:2.0.1
like it mentioned in Docs

Then it throws below error, it needs 2 more args which was not mentioned in Docs
Unable to detect Confluent Platform installation. Specify --component-dir and --worker-configs explicitly.

Error: Invalid options or arguments

and Ignore my second image statements , i understood that it works that way in a Distributed Mode

As the error output says, you need to provide the locations where the connect worker properties are stored and where you want the connector to be downloaded.

Similar question - apache kafka - Use Confluent Hub without Confluent Platform installation - Stack Overflow

1 Like