# S3 Sink Connector writes only 3 messages for each file

**URL:** <https://forum.confluent.io/t/s3-sink-connector-writes-only-3-messages-for-each-file/4570>\
**Category:** Kafka Connect\
**Created:** [24 March 2022 14:16 UTC](https://forum.confluent.io/t/s3-sink-connector-writes-only-3-messages-for-each-file/4570 "2022-03-24T14:16:53Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![shortcode](https://avatars.discourse-cdn.com/v4/letter/s/eada6e/32.png) [@shortcode](https://forum.confluent.io/u/shortcode)\
**Post date:** [24 March 2022 14:16 UTC](https://forum.confluent.io/t/s3-sink-connector-writes-only-3-messages-for-each-file/4570/1 "2022-03-24T14:16:54Z")

</div>

S3 bucket has \> 500K files and it’s hard to read them with Spark. How can I increase message count per partition? Below my config:

```auto
"config": {
    "connector.class": "S3SinkConnector",
    "topics": "xxxxxxxxxx",
    "s3.bucket.name": "xxxxxxxxxxxx",
    "s3.part.size":"26214400",
    "format.class": "io.confluent.connect.s3.format.parquet.ParquetFormat",
    "parquet.codec": "snappy",
    "storage.class": "io.confluent.connect.s3.storage.S3Storage",
    "flush.size": "10000",
    "s3.region": "xxxxxxxx",
    "tasks.max": "3",
    "partitioner.class": "io.confluent.connect.storage.partitioner.TimeBasedPartitioner",
    "locale": "en-US",
    "timezone": "UTC",
    "rotate.schedule.interval.ms": "3600000",
    "rotate.interval.ms": "60000",
    "path.format": "YYYY-MM-dd",
    "timestamp.extractor": "Record",
    "transforms": "insertTS,formatTS, ReplaceField",
    "transforms.insertTS.type": "org.apache.kafka.connect.transforms.InsertField$Value",
    "transforms.insertTS.timestamp.field": "messageTS",
    "transforms.formatTS.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value",
    "transforms.formatTS.format": "yyyy-MM-dd HH:mm:ss:SSS",
    "transforms.formatTS.field": "messageTS",
    "transforms.formatTS.target.type": "string",
    "transforms.ReplaceField.type": "org.apache.kafka.connect.transforms.ReplaceField$Value",
    "transforms.ReplaceField.blacklist": "xxxxxxxxxxxxxx"
  }

```

---

<div class="post-metadata">

**Author:** ![rmoff](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/rmoff/32/3_2.png) [@rmoff](https://forum.confluent.io/u/rmoff)\
**Post date:** [24 March 2022 16:07 UTC](https://forum.confluent.io/t/s3-sink-connector-writes-only-3-messages-for-each-file/4570/2 "2022-03-24T16:07:59Z")

</div>

> [@shortcode](#):
>
> ```
> "rotate.interval.ms": "60000",
> 
> ```

You’re creating a new file every minute (60000 ms = 60 seconds = 1 minute). That’s why you’ve got so many files.

---

<div class="post-metadata">

**Author:** ![shortcode](https://avatars.discourse-cdn.com/v4/letter/s/eada6e/32.png) [@shortcode](https://forum.confluent.io/u/shortcode)\
**Post date:** [25 March 2022 08:54 UTC](https://forum.confluent.io/t/s3-sink-connector-writes-only-3-messages-for-each-file/4570/3 "2022-03-25T08:54:50Z")

</div>

If I try to increase let say 5 min. I got error: `java.lang.OutOfMemoryError: Java heap space`

---

<div class="post-metadata">

**Author:** ![shortcode](https://avatars.discourse-cdn.com/v4/letter/s/eada6e/32.png) [@shortcode](https://forum.confluent.io/u/shortcode)\
**Post date:** [31 March 2022 13:28 UTC](https://forum.confluent.io/t/s3-sink-connector-writes-only-3-messages-for-each-file/4570/4 "2022-03-31T13:28:02Z")

</div>

had to increase connector’s heap size. now working fine.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/confluentcommunity/original/1X/c49438c90c9df282e9996fdf6971be890c71b65a.svg) [@system](https://forum.confluent.io/u/system)\
**Post date:** [7 April 2022 13:28 UTC](https://forum.confluent.io/t/s3-sink-connector-writes-only-3-messages-for-each-file/4570/5 "2022-04-07T13:28:09Z")

</div>

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.