How to create new workers for Kafka-Connect Distributed Mode?

Hi. I’ve just installed Confluent and everything was fine. Now I want to do a bit of an advanced thing, in particular, I want to add more “workers” in distributed mode.
I followed this guide and run this command
“bin/connect-distributed”, then I added a connector through REST API, and everything was fine.
Now how to add a new worker and run Kafka Connect with 2 workers instead?
In the docs, it said that " Note that if you run many distributed workers on one host machine for development and testing, the listeners configuration property must be unique for each worker. This is the port the REST interface listens on for HTTP requests"

When I checked the file, I can’t find any “listeners” related options. The only thing I found is
"# These are provided to inform the user about the presence of the REST host and port configs

Hostname & Port for the REST API to listen on. If this is set, it will bind to the interface used to listen to requests.

The Hostname & Port that will be given out to other workers to connect to i.e. URLs that are routable from other servers.

So what should I change to add a new worker? I should copy and paste to make 2 files, right? The command now should be bin/connect-distributed,, right?

Hi @quang the property you’ll want to set is listeners doumented in the common worker configuration section here. So the quickest way to test this is to create two worker properties files that have the same (so that they’re in the same connect cluster), and set this in one:


and this in the other:


Then start them with separate commands:


Now you can instantiate a connector on either endpoint, i.e.,

curl -s -H "Content-Type: application/json" -X POST -d @my-connector.json http://localhost:8083/connectors/


curl -s -H "Content-Type: application/json" -X POST -d @spool.json http://localhost:8084/connectors/

And you can play with scenarios like stopping a worker with running tasks and observe the tasks move to the other worker.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.