Hi Team,
We are trying to create kafka connect s3 connector using dockerfile, below are the dockerfile and docker-compose file. My kafka cluster is hosted in AWS (managed streaming kafka). when i start my docker container i am facing below error.
my connectivity seems good from my docker instance to mskCluster, tested the same with telnet.
Error while getting broker list.
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1680690123968, tries=1, nextAllowedTryMs=1680690124088) timed out at 1680690123988 after 1 attempt(s)
** at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)**
** at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)**
** at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)**
** at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)**
** at io.confluent.admin.utils.ClusterStatus.isKafkaReady(ClusterStatus.java:147)**
** at io.confluent.admin.utils.cli.KafkaReadyCommand.main(KafkaReadyCommand.java:149)**
Dockerfile
FROM confluentinc/cp-kafka-connect-base:6.2.10
RUN echo “\nInstalling all required connectors…\n” &&
confluent-hub install --no-prompt confluentinc/kafka-connect-s3:latest
docker-compose file
version: ‘2’
services:
kafka-connect:
image: sinkconnector:latest
environment:
CONNECT_BOOTSTRAP_SERVERS: “{bootstrap_connection} to communicate with mskCluster”
CONNECT_REST_ADVERTISED_HOST_NAME: “localhost”
CONNECT_PLUGIN_PATH: “/usr/share/java,/usr/share/confluent-hub-components,/opt/connectors”
CONNECT_CONFIG_STORAGE_TOPIC: “_connect-configs”
CONNECT_OFFSET_STORAGE_TOPIC: “_connect-offsets”
CONNECT_STATUS_STORAGE_TOPIC: “_connect-status”
CONNECT_GROUP_ID: “kafka-connect-s3”
CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: “1”
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: “1”
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: “1”
CONNECT_KEY_CONVERTER: “org.apache.kafka.connect.storage.StringConverter”
CONNECT_VALUE_CONVERTER: “io.confluent.connect.s3.format.json.JsonFormat”
CONNECT_INTERNAL_KEY_CONVERTER: “org.apache.kafka.connect.json.JsonConverter”
CONNECT_INTERNAL_VALUE_CONVERTER: “org.apache.kafka.connect.json.JsonConverter”
volumes:
- ./config:/opt/connectors/config