# Incresing the number of tasks for an S3 sink connector

**URL:** <https://forum.confluent.io/t/incresing-the-number-of-tasks-for-an-s3-sink-connector/1342>\
**Category:** Kafka Connect\
**Created:** [30 March 2021 13:55 UTC](https://forum.confluent.io/t/incresing-the-number-of-tasks-for-an-s3-sink-connector/1342 "2021-03-30T13:55:17Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![shlomi](https://avatars.discourse-cdn.com/v4/letter/s/cab0a1/32.png) [@shlomi](https://forum.confluent.io/u/shlomi)\
**Post date:** [30 March 2021 13:55 UTC](https://forum.confluent.io/t/incresing-the-number-of-tasks-for-an-s3-sink-connector/1342/1 "2021-03-30T13:55:18Z")

</div>

Hey all,

I have a source connector (debezium) that fetch data from Postgres into Kafka. In addition, I have a S3 sink that writes that data from Kafka into S3. When I create the S3 sink connector I set the number of tasks to 1.  
Now I want to allow more tasks / threads / workers, in order to lower the lag from Kafka to S3, I have modified the connector, to have 17 tasks and I can see that Kafka connect list 17 tasks, but while looking at the logs it still looks like the first task (with id=0) it doing all the work.

Any ideas on how to fix it?

currently I’m running a single Docker container on a single EC2 instance, running both connectors (debezium and S3).

Thanks.

---

<div class="post-metadata">

**Author:** ![rmoff](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/rmoff/32/3_2.png) [@rmoff](https://forum.confluent.io/u/rmoff)\
**Post date:** [30 March 2021 14:31 UTC](https://forum.confluent.io/t/incresing-the-number-of-tasks-for-an-s3-sink-connector/1342/2 "2021-03-30T14:31:42Z")

</div>

How many partitions does the topic have? IIRC the connector can parallelise operation over multiple partitions only.

---

<div class="post-metadata">

**Author:** ![shlomi](https://avatars.discourse-cdn.com/v4/letter/s/cab0a1/32.png) [@shlomi](https://forum.confluent.io/u/shlomi)\
**Post date:** [30 March 2021 14:50 UTC](https://forum.confluent.io/t/incresing-the-number-of-tasks-for-an-s3-sink-connector/1342/3 "2021-03-30T14:50:01Z")

</div>

That makes sense. I have 17 topics and each one has only 1 partition, so I thought that each task can process a different topic / partition (in my case).

---

<div class="post-metadata">

**Author:** ![rmoff](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/rmoff/32/3_2.png) [@rmoff](https://forum.confluent.io/u/rmoff)\
**Post date:** [30 March 2021 15:29 UTC](https://forum.confluent.io/t/incresing-the-number-of-tasks-for-an-s3-sink-connector/1342/4 "2021-03-30T15:29:15Z")

</div>

Yes, I would expect to see it do that. Can you share your connector config?

---

<div class="post-metadata">

**Author:** ![shlomi](https://avatars.discourse-cdn.com/v4/letter/s/cab0a1/32.png) [@shlomi](https://forum.confluent.io/u/shlomi)\
**Post date:** [30 March 2021 17:41 UTC](https://forum.confluent.io/t/incresing-the-number-of-tasks-for-an-s3-sink-connector/1342/5 "2021-03-30T17:41:20Z")

</div>

sorry wrong thread, will update with the connector config later. Thanks

---

<div class="post-metadata">

**Author:** ![shlomi](https://avatars.discourse-cdn.com/v4/letter/s/cab0a1/32.png) [@shlomi](https://forum.confluent.io/u/shlomi)\
**Post date:** [31 March 2021 07:09 UTC](https://forum.confluent.io/t/incresing-the-number-of-tasks-for-an-s3-sink-connector/1342/6 "2021-03-31T07:09:01Z")

</div>

```
{
"config": {
    "behavior.on.null.values": "ignore",
    "connector.class": "io.confluent.connect.s3.S3SinkConnector",
    "flush.size": "1000000",
    "format.class": "io.confluent.connect.s3.format.parquet.ParquetFormat",
    "locale": "US",
    "partition.duration.ms": "31556952000",
    "partitioner.class": "io.confluent.connect.storage.partitioner.TimeBasedPartitioner",
    "path.format": "'year'=YYYY/",
    "rotate.schedule.interval.ms": "180000",
    "s3.bucket.name": "s3_sink_data",
    "s3.region": "eu-west-1",
    "storage.class": "io.confluent.connect.s3.storage.S3Storage",
    "tasks.max": "17",
    "timezone": "UTC",
    "topics.dir": "db_data",
    "topics.regex": "db_data.public.*"
},
"name": "db-to-s3-sink"
}

```

could it be that since I created the connector with only 1 task all the existing partitions have been “assigned” to the first task?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/confluentcommunity/original/1X/c49438c90c9df282e9996fdf6971be890c71b65a.svg) [@system](https://forum.confluent.io/u/system)\
**Post date:** [30 April 2021 07:09 UTC](https://forum.confluent.io/t/incresing-the-number-of-tasks-for-an-s3-sink-connector/1342/7 "2021-04-30T07:09:35Z")

</div>

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.