Hi,
I’m using the s3 sink connector with the following configs:
"partitioner.class":"io.confluent.connect.storage.partitioner.TimeBasedPartitioner",
"locale":"en",
"path.format":"'date'=YYYY-MM-dd",
"partition.duration.ms":"86400000",
"rotate.interval.ms":"3600000",
"timestamp.extractor":"RecordField",
"timestamp.field": "dateCreated"
and also:
"flush.size": "100"
My issue is that in s3 I see files are keeping modifying in past partitions, this prevents me to run daily jobs (say at 00:00 UTC on the past partitions)
What might be the reason? how can I make sure all data is flushed at 00:00?