I have a source connector (debezium) that fetch data from Postgres into Kafka. In addition, I have a S3 sink that writes that data from Kafka into S3. When I create the S3 sink connector I set the number of tasks to 1.
Now I want to allow more tasks / threads / workers, in order to lower the lag from Kafka to S3, I have modified the connector, to have 17 tasks and I can see that Kafka connect list 17 tasks, but while looking at the logs it still looks like the first task (with id=0) it doing all the work.
Any ideas on how to fix it?
currently I’m running a single Docker container on a single EC2 instance, running both connectors (debezium and S3).