Kafka connect don't start on Docker

Kafka Connect
1

I am a newbie to kafka and following the kafka connect 101 exercises.
When i try to start connect on a container i ger the error:

INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:57)
INFO [Producer clientId=kc101-connect–offsets] Cluster ID: lkc-j8j5ww (org.apache.kafka.clients.Metadata:287)
INFO [Consumer clientId=kc101-connect–offsets, groupId=kc101-connect] Cluster ID: lkc-j8j5ww (org.apache.kafka.clients.Metadata:287)
ERROR [Worker clientId=connect-1, groupId=kc101-connect] Uncaught exception in herder work thread, exiting: (org.apache.kafka.connect.runtime.distributed.DistributedHerder:377)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [_kc101-connect-offsets]
INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:67)
2023-10-25 15:39:37 [2023-10-25 19:39:37,785] INFO Stopped http_connect-18083@5d514625{HTTP/1.1, (http/1.1)}{connect-1:8083} (org.eclipse.jetty.server.AbstractConnector:383)

This is what i have in the docker file:

image: confluentinc/cp-server-connect:7.5.1
environment:
CONNECT_BOOTSTRAP_SERVERS: $BOOTSTRAP_SERVERS
CONNECT_GROUP_ID: “kc101-connect”
CONNECT_CONFIG_STORAGE_TOPIC: “_kc101-connect-configs”
CONNECT_OFFSET_STORAGE_TOPIC: “_kc101-connect-offsets”
CONNECT_STATUS_STORAGE_TOPIC: “_kc101-connect-status”
CONNECT_REPLICATION_FACTOR: 1
CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
CONNECT_KEY_CONVERTER: “org.apache.kafka.connect.storage.StringConverter”

  # Confluent Schema Registry for Kafka Connect
  CONNECT_VALUE_CONVERTER: "io.confluent.connect.avro.AvroConverter"
  CONNECT_VALUE_CONVERTER_SCHEMAS_ENABLE: "true"
  CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: $SCHEMA_REGISTRY_URL
  CONNECT_VALUE_CONVERTER_BASIC_AUTH_CREDENTIALS_SOURCE: $BASIC_AUTH_CREDENTIALS_SOURCE
  CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO: $SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO

  CONNECT_REST_ADVERTISED_HOST_NAME: "connect-1"
  CONNECT_LISTENERS: http://connect-1:8083
  CONNECT_PLUGIN_PATH: "/usr/share/java,/usr/share/confluent-hub-components"
  CONNECT_LOG4J_APPENDER_STDOUT_LAYOUT_CONVERSIONPATTERN: "[%d] %p %X{connector.context}%m (%c:%L)%n"
  CONNECT_LOG4J_ROOT_LOGLEVEL: INFO
  CONNECT_LOG4J_LOGGERS: 'org.apache.kafka.connect.runtime.rest=WARN,org.reflections=ERROR'

  # Confluent Cloud config
  CONNECT_REQUEST_TIMEOUT_MS: "20000"
  CONNECT_RETRY_BACKOFF_MS: "500"
  CONNECT_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: "https"

  # Connect worker
  CONNECT_SECURITY_PROTOCOL: SASL_SSL
  CONNECT_SASL_JAAS_CONFIG: $SASL_JAAS_CONFIG
  CONNECT_SASL_MECHANISM: PLAIN

  # Connect producer
  CONNECT_PRODUCER_SECURITY_PROTOCOL: SASL_SSL
  CONNECT_PRODUCER_SASL_JAAS_CONFIG: $SASL_JAAS_CONFIG
  CONNECT_PRODUCER_SASL_MECHANISM: PLAIN

  # Connect consumer
  CONNECT_CONSUMER_SECURITY_PROTOCOL: SASL_SSL
  CONNECT_CONSUMER_SASL_JAAS_CONFIG: $SASL_JAAS_CONFIG
  CONNECT_CONSUMER_SASL_MECHANISM: PLAIN
2

Hi @lizard,

Likely this is due to Step 14 command being incomplete. It should be:

confluent kafka client-config create java  --api-key <KAFKA API KEY> --api-secret <KAFKA SECRET> --schema-registry-api-key <SR API KEY> --schema-registry-api-secret <SR SECRET>

Before starting the containers, double check that the following environment variables referenced in docker-compose.yml are populated with your credentials:

$ echo $SASL_JAAS_CONFIG
org.apache.kafka.common.security.plain.PlainLoginModule required username='****************' password='*********************************************************';

$ echo $SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO
****************:************************************************************

HTH,
Dave

3

Thanks for the answer :slight_smile:

  1. I followed the instructions and generated a new java.config file. It was the same as the one i have before.
  2. Checked the env variables and they are correct.
    The problem persisted

Then pocking around files i noticed the following in the file connect-ccloud.delta
replication.factor=3
config.storage.replication.factor=3
offset.storage.replication.factor=3
status.storage.replication.factor=3

I changed similar variables in the doker compose file to have the same value:
CONNECT_REPLICATION_FACTOR: 3
CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 3
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 3
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 3

then re-started my container and my connect worker started successfully. I can see the connect compacted topics created in the confluent cloud cluster.

It worked but is still not clear for me why. What was the problem?

4

That’s odd – I’m not sure. If you stop the container and go back to replication factor 1, does it fail with the auth exception?

The original docker compose starts up ok on my end, that’s why I thought it was an environment variable problem.

If the container starts up ok if you go back to replication factor 1, maybe the exception was due to credentials not having propagated. How long was it between creating API keys and starting the Connect worker container?

Dave

5

Yes it is odd.
I put the replication factors back to 1 i was able to reproduce the problem but now i get better error messages:
2023-10-27 15:21:58 Running in a “–no-prompt” mode
2023-10-27 15:22:04 javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2023-10-27 15:22:04
2023-10-27 15:22:04 Error: Unknown error
2023-10-27 15:22:04 Launching Kafka Connect worker

2023-10-27 15:24:22 [2023-10-27 19:24:22,042] ERROR [Worker clientId=connect-1, groupId=lil_kc101-connect] Uncaught exception in herder work thread, exiting: (org.apache.kafka.connect.runtime.distributed.DistributedHerder:376)
2023-10-27 15:24:22 org.apache.kafka.connect.errors.ConnectException: Error while attempting to create/find topic(s) ‘lil_kc101-connect-offsets’
2023-10-27 15:24:22 Caused by: org.apache.kafka.common.errors.PolicyViolationException: Topic replication factor must be 3

Change then back to 3 it is back to working.
I can continue working on setting a Splunk Sink connector :slight_smile:

Thanks very much for your help

6

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.