Currently I have a distributed worker running on linux and sending json to ES using the ES connector. If I wanted to make this highly available I would create another linux server running the distributed worker. Now if i run the distributed worker on both machines with an ES connector set up for b…

It sounds like you’ve not formed the workers into a Kafka Connect group. When configured correctly the work is split across tasks which execute on one, or the other, worker - not both (unless there is more than one task). Have a look at Common mistakes made when configuring multiple Kafka Connect w…

Great. Thank you for that information. I have seem to get a lot further now. I now see the leader URL of the IP of one of the workers in both of the workers logs. When I send data though it does not duplicate but it seems like it only send to one of the workers. When I use this rest.advertised.ho…

I think I got it. The tasks.max was set to 1. Should this be set to a specific number?

In a Kafka Connect sink, the tasks are essentially consumer threads and receive partitions to read from. If you have 10 partitions and have tasks.max set to 5, each task will receive 2 partitions to read from and track the offsets. If you have configured tasks.max to a number above the partition cou…

Thank you. So if I have two workers what is recommended on the amount of tasks to set. Is it better to have more tasks then partitions?

My understanding is that there’s no point setting more tasks than partitions.

Hi, when running workers in Distributed mode, GROUP_ID: 4 is important one, it should match to the first worker and is required in order to determine the Cluster that the worker will be part off.

Great. Thanks everyone. I think I am good now. I will continue to do some more testing. If I run into any issues I will let you know.

Kafka Connect Distributed Worker

Kafka Connect

Topic		Replies	Views
Different kafka cluster and different connect cluster Self-Managed Connectors	11	4770	11 June 2021
Parallelism and Load Balancing in Distributed Kafka Connect Deployment Kafka Connect	0	231	19 June 2024
✍️ Common mistakes made when configuring multiple Kafka Connect workers Kafka Connect	0	3136	17 November 2020
Kafka connector, workers Kafka Connect	1	2151	12 May 2023
Kafka Connect Elasticsearch sink stops sending records Self-Managed Connectors	13	5140	12 February 2021

Kafka Connect Distributed Worker

Related topics