How to create new workers for Kafka-Connect Distributed Mode?

Kafka Connect
1

Hi. I’ve just installed Confluent and everything was fine. Now I want to do a bit of an advanced thing, in particular, I want to add more “workers” in distributed mode.
I followed this guide and run this command
“bin/connect-distributed worker.properties”, then I added a connector through REST API, and everything was fine.
Now how to add a new worker and run Kafka Connect with 2 workers instead?
In the docs, it said that " Note that if you run many distributed workers on one host machine for development and testing, the listeners configuration property must be unique for each worker. This is the port the REST interface listens on for HTTP requests"

When I checked the worker.properties file, I can’t find any “listeners” related options. The only thing I found is
"# These are provided to inform the user about the presence of the REST host and port configs

Hostname & Port for the REST API to listen on. If this is set, it will bind to the interface used to listen to requests.

#rest.host.name=0.0.0.0
#rest.port=8083

The Hostname & Port that will be given out to other workers to connect to i.e. URLs that are routable from other servers.

#rest.advertised.host.name=0.0.0.0
#rest.advertised.port=8083"

So what should I change to add a new worker? I should copy and paste to make 2 worker.properties files, right? The command now should be bin/connect-distributed worker1.properties, worker2.properties, right?

2

Hi @quang the property you’ll want to set is listeners doumented in the common worker configuration section here. So the quickest way to test this is to create two worker properties files that have the same group.id (so that they’re in the same connect cluster), and set this in one:

listeners=http://0.0.0.0:8083

and this in the other:

listeners=http://0.0.0.0:8084

Then start them with separate commands:

./bin/connect-distributed worker2.properties
./bin/connect-distributed worker1.properties

Now you can instantiate a connector on either endpoint, i.e.,

curl -s -H "Content-Type: application/json" -X POST -d @my-connector.json http://localhost:8083/connectors/

or

curl -s -H "Content-Type: application/json" -X POST -d @spool.json http://localhost:8084/connectors/

And you can play with scenarios like stopping a worker with running tasks and observe the tasks move to the other worker.

3

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.