Kafka-connect takes too long to spin up

Greetings,
Im using similar docker compose file as this , but kafka connect takes too long to spin up. As you can see Im waiting for 6 mins and it still starting. It seems to be stuck. .

connect part in compose file:
  kafka-connect-01:
image: confluentinc/cp-kafka-connect:5.4.0
container_name: kafka-connect-01
depends_on:
  - zookeeper
  - broker
  - schema-registry
ports:
  - 8083:8083
environment:
  CONNECT_LOG4J_APPENDER_STDOUT_LAYOUT_CONVERSIONPATTERN: "[%d] %p %X{connector.context}%m (%c:%L)%n"
  CONNECT_BOOTSTRAP_SERVERS: "broker:29092"
  CONNECT_REST_PORT: 8083
  CONNECT_REST_ADVERTISED_HOST_NAME: "kafka-connect-01"
  CONNECT_GROUP_ID: compose-connect-group
  CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
  CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
  CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
  CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
  CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
  CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.storage.StringConverter
  CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
  CONNECT_INTERNAL_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
  CONNECT_INTERNAL_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
  CONNECT_LOG4J_ROOT_LOGLEVEL: "ERROR"
  CONNECT_LOG4J_LOGGERS: "org.apache.kafka.connect.runtime.rest=WARN,org.reflections=ERROR"
  CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: "1"
  CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: "1"
  CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: "1"
  CONNECT_PLUGIN_PATH: "/usr/share/java"

#   - /my/local/folder/with/jdbc-driver.jar:/usr/share/java/kafka-connect-jdbc/jars/
command:
  - /bin/bash
  - -c
  - |
    # JDBC Drivers
    # ------------
    # MySQL
    cd /usr/share/java/kafka-connect-jdbc/
    # See https://dev.mysql.com/downloads/connector/j/
    curl https://cdn.mysql.com/Downloads/Connector-J/mysql-connector-java-8.0.23.tar.gz | tar xz 
    /etc/confluent/docker/run &
    sleep infinity

Btw, in your demo that I linked above, you have this part for kafka connect:

sleep infinity &
/etc/confluent/docker/run

should not it be the opposite(sleep being the last one)?
connect logs:

===> ENV Variables …
ALLOW_UNSIGNED=false
COMPONENT=kafka-connect
CONFLUENT_DEB_VERSION=1
CONFLUENT_PLATFORM_LABEL=
CONFLUENT_VERSION=5.4.0
CONNECT_BOOTSTRAP_SERVERS=broker:29092
CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=1
CONNECT_CONFIG_STORAGE_TOPIC=docker-connect-configs
CONNECT_GROUP_ID=compose-connect-group
CONNECT_INTERNAL_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
CONNECT_INTERNAL_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
CONNECT_KEY_CONVERTER=org.apache.kafka.connect.storage.StringConverter
CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL=http://schema-registry:8081
CONNECT_LOG4J_APPENDER_STDOUT_LAYOUT_CONVERSIONPATTERN=[%d] %p %X{connector.context}%m (%c:%L)%n
CONNECT_LOG4J_LOGGERS=org.apache.kafka.connect.runtime.rest=WARN,org.reflections=ERROR
CONNECT_LOG4J_ROOT_LOGLEVEL=ERROR
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=1
CONNECT_OFFSET_STORAGE_TOPIC=docker-connect-offsets
CONNECT_PLUGIN_PATH=/usr/share/java
CONNECT_REST_ADVERTISED_HOST_NAME=kafka-connect-01
CONNECT_REST_PORT=8083
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=1
CONNECT_STATUS_STORAGE_TOPIC=docker-connect-status
CONNECT_VALUE_CONVERTER=org.apache.kafka.connect.storage.StringConverter
CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL=http://schema-registry:8081
CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
HOME=/root
HOSTNAME=59b8cd84053f
KAFKA_ADVERTISED_LISTENERS=
KAFKA_VERSION=
KAFKA_ZOOKEEPER_CONNECT=
LANG=C.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/usr/share/java/kafka-connect-jdbc
PYTHON_PIP_VERSION=8.1.2
PYTHON_VERSION=2.7.9-1
SCALA_VERSION=2.12
SHLVL=2
ZULU_OPENJDK_VERSION=8=8.38.0.13
_=/usr/bin/env
===> User
uid=0(root) gid=0(root) groups=0(root)
===> Configuring …
===> Running preflight checks …
===> Check if Kafka is healthy …
===> Launching …
===> Launching kafka-connect …

after 16 mins, status is unhealthy
The log is exactly the same.

Hi Otto,

How much memory do you have allocated to docker? Sometimes this can be an issue.

Is the log you’ve quoted from docker logs kafka-connect?

Good question, but the sleep is forked to the background (&) so the order in this case is not incorrect.

Thanks Robin, I increased the resources for docker and it seems to work as expected now.
I also have some concerns, I will list here:
In the demo, for control -center we have

    sleep 120  
    /etc/confluent/docker/run
  1. Since we declare broker as a dependency, do we have to still wait 2 mins manually? I excluded it and seems to work.

  2. Isnt, CONNECT_INTERNAL_VALUE_CONVERTER and CONNECT_INTERNAL_KEY_CONVERTER deprecated?

  3. When we specify CONNECT_PLUGIN_PATH, will connect search for the plugin recursively? For example, if we have /usr/share/java will it search in every folder recursively?

Thanks

The dependency is just for Docker Compose in the order in which it initiates each container. It doesn’t make it wait until the container has finished booting up. The two minute wait is a bit of a hacky-but-good-enough approach to make sure that the broker has finished starting before the control center tries to use it

Yes. Feel free to submit a PR to the repository to remove these :slight_smile:

Yes, I think it must do (although I’m not 100% sure if there’s a limit on the level of recursion). For example, here I would set plugin.path=/opt/connectors

/opt/connectors
└── jcustenborder-kafka-connect-spooldir
    ├── doc
    │   ├── LICENSE
    │   └── README.md
    ├── etc
        …
    ├── lib
        …
    │   ├── javassist-3.21.0-GA.jar
    │   ├── jsr305-3.0.2.jar
    │   ├── kafka-connect-spooldir-2.0.43.jar
    │   ├── listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
        …
    └── manifest.json

Since plugin.path can be a comma-separated list of paths, I would strongly suggest you set it to point at the folder(s) that contains the connectors (rather than just relying on a broad search from a high level in the file system) simply because it makes it easier to maintain (anyone else inspecting the config will know where the connectors are supposed to be).