S3SinkConnector can't find only a specific S3 bucket

Hi there,

We have the S3SinkConnector installed in our Confluent Kafka Connect and while using about 30 connectors that are exports data to the AWS S3 bucket(“test-bucket/a”) are running.

“test-bucket/a” bucket access is possibe with AWS CLI, and running connectors also operates normally.
But it is impossible to create a new connector only in the “test-bucket/a” as shown below.

  1. “s3.bucket.name”: “test-bucket/a”
  • The existing connector is operating normally, But failed on new creation.
  1. “s3.bucket.name”: “test-bucket/b”
  • Creation successful.
  1. “s3.bucket.name”: “test-bucket”
  • Creation successful.

When the task starts running we got this error.

org.apache.kafka.connect.errors.ConnectException: Non-existent S3 bucket: test-bucket/a
io.confluent.connect.s3.S3SinkTask.start(S3SinkTask.java:113)

What is the problem?

Thanks.

Last I checked, buckets cannot contain slashes. I assume you meant to use topics.dir to set up a prefix within the test-bucket bucket?

AWS S3 full path looks like below.
Amazon S3 > bucket > test-bucket > a/ > topics/ > topicname/ > yyyymmdd/ > .json files

It was going well, but why isn’t it working all of a sudden?
It works well for other directory(test-bucket/b) in the test-bucket.

Like I said, it seems you want s3.bucket.name=test-bucket with topics.dir=a/topics, not "s3.bucket.name=test-bucket/a"

The problem was resolved using the topics.dir option. Thank you very much.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.