Kafka Connect request timeout

Kafka Connect
1

I’ve standalone connect and it will consume from external Kafka. connection between servers tested over 9092 port and works.

this line works fine. and returns : 404 connector not exist
curl -X GET connectUrl:8084/connectors/connectorName/status | jq

curl -d @"/etc/kafka/s3-aa-mapping-sink.json" -H "Content-Type: application/json" -X POST connectUrl:8084/connectors

returns 500 {"error_code":500,"message":"Request timed out"}

connect log:

[2021-03-25 15:20:13,971] WARN [Producer clientId=producer-3] Got error produce response with correlation id 13934 on topic-partition connect-award-article-mapping-configs-0, retrying (2147469731 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender:595)
[2021-03-25 15:20:14,107] WARN [Producer clientId=producer-3] Got error produce response with correlation id 13935 on topic-partition connect-award-article-mapping-configs-0, retrying (2147469730 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender:595)
[2021-03-25 15:20:14,243] WARN [Producer clientId=producer-3] Got error produce response with correlation id 13936 on topic-partition connect-award-article-mapping-configs-0, retrying (2147469729 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender:595)
2

Can you share your connector config please?

3 
{

  "name": "award-article-mapping",

  "config": {

    "connector.class": "S3SinkConnector",

    "topics": "award-article.link",

    "s3.bucket.name": "*********t",

    "format.class": "io.confluent.connect.s3.format.parquet.ParquetFormat",

    "parquet.codec": "snappy",

    "storage.class": "io.confluent.connect.s3.storage.S3Storage",

    "flush.size": "2147483647",

    "s3.region": "us-east-2",

    "tasks.max": "3",

    "partition.duration.ms": "3600000",

    "s3.credentials.provider.sts.role.arn": "********",

    "s3.credentials.provider.class": "io.confluent.connect.s3.auth.AwsAssumeRoleCredentialsProvider",

    "s3.credentials.provider.sts.role.session.name": "kafka-connect-s3-sink",

    "s3.credentials.provider.sts.role.external.id": "kafka-connect-s3-sink",

    "partitioner.class": "io.confluent.connect.storage.partitioner.TimeBasedPartitioner",

    "locale": "en-US",

    "timezone": "UTC",

    "rotate.schedule.interval.ms": "3600000",

    "rotate.interval.ms": "3600000",

    "path.format": "YYYY-MM-dd",

    "timestamp.extractor": "Record"

  }

}
4

From your error:

Are you definitely using standalone mode? This looks like a distributed worker config topic

5

Ahh, yeah it’s distributed. What should I do? I’ve used existing terraform and it was setup whole set Kafka, schema registry, connect, zookeeper.

I just changed bootstrap servers from local to external brokers (3).

In production it’s better to be distributed right?

6

Distributed mode is generally recommended unless you need execution locality (e.g. processing log files / receiving inbound connections on a given host for syslog data etc).

See: Should you run Kafka Connect in Distributed or Standalone mode? - DEV Community 👩‍💻👨‍💻

Regarding the error, my guess is the internal topic has been created with three replicas but you don’t have that many brokers available. Depending on your situation you may want to run with fewer replicas (e.g. on a sandbox machine and just one broker) or bring up more brokers (if there should be three).

See Worker Configuration Properties | Confluent Documentation

7

So this error caused timed out?

8

Yes, if your workers are not correctly configured there’ll not be able to save new connector details to the config topic, and i would expect this kind of error.

9

Updating worker settings helped. Now connectors running but s3 bucket empty. There is no error message on connect logs. Where should I look?

10

If there’s no errors and no data in S3 then it suggests one of:

  • Kafka Connect is reading from a different topic
  • No [new] data in the source topic
  • Data is being written to an S3 bucket other than the one you’re looking in
11

Yeah. there was no messages :slight_smile:
thanks Robin.

12

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.