Tried creating s3 connector aiming to copy the content of kafka topic to S3 bucket
connector also got created.
[kafka-fast-cp-kafka-connect-6c85cc9498-7lsfh appuser]# curl -X GET http://localhost:8083/connectors
[“my-s3-sink”]
Used the below rest endpoint to create the connector but still couldnt connect to S3 bucket yet. Getting amazon s3 location constraint specific exception.
Error reference:
org.apache.kafka.connect.errors.ConnectException: com.amazonaws.services.s3.model.AmazonS3Exception: The eu-west-1 location constraint is incompatible for the region specific endpoint this request was sent to. (Service: Amazon S3; Status Code: 400; Error Code: IllegalLocationConstraintException; Request ID: 1G186EX425ARFWXG; S3 Extended Request ID: bqEXZXBB+f0mYqJLYSMWXe7RQVtt0M7hMSG9jw1uW7BuK8KniFt+qyoOmi1yqcyZek4GdVtdMl0=; Proxy: null), S3 Extended Request ID: bqEXZXBB+f0mYqJLYSMWXe7RQVtt0M7hMSG9jw1uW7BuK8KniFt+qyoOmi1yqcyZek4GdVtdMl0=
at io.confluent.connect.s3.S3SinkTask.start(S3SinkTask.java:138)
at org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:305)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:196)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The eu-west-1 location constraint is incompatible for the region specific endpoint this request was sent to. (Service: Amazon S3; Status Code: 400; Error Code: IllegalLocationConstraintException; Request ID: 1G186EX425ARFWXG; S3 Extended Request ID: bqEXZXBB+f0mYqJLYSMWXe7RQVtt0M7hMSG9jw1uW7BuK8KniFt+qyoOmi1yqcyZek4GdVtdMl0=; Proxy: null), S3 Extended Request ID: bqEXZXBB+f0mYqJLYSMWXe7RQVtt0M7hMSG9jw1uW7BuK8KniFt+qyoOmi1yqcyZek4GdVtdMl0=
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
s3.region already set to HK(ap-east-1). Still should i need to hardcode this anywhere? Kindly help in resolving the issue
curl used for creation of sink connector:
curl -X POST -H ‘Content-Type: application/json’ --data ’
{“name”: “my-s3-sink”,
“config”: {“connector.class”:“io.confluent.connect.s3.S3SinkConnector”,
“tasks.max”:“1”,
“topics”:“fast-id-01batchinitoutblue-eventstore”,
“s3.region”:“ap-east-1”,
“s3.bucket.name”: “022376188946-s3-access-logs”,
“s3.part.size”:“5242880”,
“flush.size”:“1”,
“storage.class”:“io.confluent.connect.s3.storage.S3Storage”,
“format.class”: “io.confluent.connect.s3.format.json.JsonFormat”,
“partitioner.class”:“io.confluent.connect.storage.partitioner.DefaultPartitioner”,
“schema.compatibility”:“NONE”,
"transforms ": “AddMetadata”,
“transforms.AddMetadata.type”: “org.apache.kafka.connect.transforms.InsertField$Value”,
“transforms.AddMetadata.offset.field”: “_offset”,
“transforms.AddMetadata.partition.field”: “_partition”}}’ http://localhost:8083/connectors