Kafka connect to s3 running but i see no data

[ighi]

Hi
I am hoping to get some help. I am quite new to kafka. So far i have installed confluent cp-all on docker on an ec2 instance. The schema registry is working. My kafka server is on aws msk. I have setup kafka connect with the aws s3 sink. it appears to be running and there are no errors in the logs but i cannot see any data being written to my bucket. Does anyone have any tips or pointers

Thanks
Ighi

[kuro]

Hello @ighi

Is your MSK using autentication? Can you share the connect worker and S3 connector configurations?

Cheers,
Brais

[ighi]

connector.class = io.confluent.connect.s3.S3SinkConnector
s3.region = <s3region>
format.class = io.confluent.connect.s3.format.bytearray.ByteArrayFormat
task.class = io.confluent.connect.s3.S3SinkTask
flush.size = 2
topics = plainone
name = plainone
aws.access.key.id = **************************
storage.class = io.confluent.connect.s3.storage.S3Storage
aws.secret.access.key = **************************
s3.bucket.name = mybucket

[ighi]

Thanks @kuro
my msk is using authentication
the s3 config is

connector.class = io.confluent.connect.s3.S3SinkConnector
s3.region = <s3region>
format.class = io.confluent.connect.s3.format.bytearray.ByteArrayFormat
task.class = io.confluent.connect.s3.S3SinkTask
flush.size = 2
topics = plainone
name = plainone
aws.access.key.id = **************************
storage.class = io.confluent.connect.s3.storage.S3Storage
aws.secret.access.key = **************************
s3.bucket.name = mybucket

I am using docker-compose on an EC2 instance - i ma not sure how to change the defaults to point to my cluster

[kuro]

@ighi , your configuration seems to be fine.

You are using personal IAM credentials autentication, does your IAM account have a policy with the permissions that allows you to use the bucket?

You can find an example of a policy for the S3 sink connector here: https://docs.confluent.io/kafka-connect-s3-sink/current/overview.html#iam-policy-for-s3

Regards,
Brais