Debezium Kafka Connect - Error Loading Plugin From Confluent Hub via Docker

Hi there,

I’ve recently tried installing the Debezium PostgreSQL Connector v1.6.0 via confluent-hub.

When the plugin is loaded, the connector fails to load, raised by loadProperties in IoUtil, per this error:

[2021-07-13 00:45:06,715] INFO Loading plugin from: /usr/share/confluent-hub-components/debezium-debezium-connector-postgresql (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2021-07-13 00:45:07,375] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed)
java.lang.ExceptionInInitializerError
at io.debezium.connector.postgresql.PostgresConnector.version(PostgresConnector.java:47)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.versionFor(DelegatingClassLoader.java:387)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.versionFor(DelegatingClassLoader.java:392)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:362)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:334)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:268)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:260)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:229)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:206)
at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:93)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80)
Caused by: java.lang.NullPointerException: inStream parameter is null
at java.base/java.util.Objects.requireNonNull(Objects.java:246)
at java.base/java.util.Properties.load(Properties.java:406)
at io.debezium.util.IoUtil.loadProperties(IoUtil.java:491)
at io.debezium.util.IoUtil.loadProperties(IoUtil.java:521)
at io.debezium.connector.postgresql.Module.<clinit>(Module.java:19)
... 12 more

We essentially have one Dockerfile, where we run the following:

FROM confluentinc/cp-kafka-connect:6.2.0

RUN confluent-hub install --no-prompt debezium/debezium-connector-postgresql:1.6.0

Everything seems to build fine, and both the zookeeper and local broker is running fine. I have noticed similar behaviour in versions 6.1.1 and 6.2.0.

Can anyone provide any input?


Steps to recreate this issue:

  1. Create a network, and run the confluentinc/cp-zookeeper:6.2.0 image with the supplied “zookeeper.env” file for environment variables:
$ docker network create --driver bridge my_network
$ docker run --platform="linux/amd64" -p "2181:2181" --name="zookeeper" --env-file=zookeeper.env --network=my_network confluentinc/cp-zookeeper:6.2.0
  1. Run the confluentinc/cp-kafka:6.2.0 image with the supplied “broker.env” file for environment variables:
$ docker run -p "29092:29092" -p "9092:9092" -p "9101:9101" --platform="linux/amd64" --name="broker" --env-file=broker.env --network=my_network confluentinc/cp-kafka:6.2.0
  1. Build a Docker image from the “Dockerfile” attached which uses the base confluentinc/cp-kafka-connect:6.2.0 . This image will install debezium-connector-postgresql:1.6.0 from Confluent Hub. Run it with the supplied “connect.env” file for environment variables:
$ docker build . --platform='linux/amd64' -t 'connect'
$ docker run -p "8083:8083" --platform="linux/amd64" --name="connect" --env-file=connect.env --network=my_network connect

Relevant .env files and the Dockerfile are hosted on this Gist.


Platform details:

  • macOS Big Sur 11.2.3 / M1
  • Docker 3.5.2 (66501)
  • Docker Engine 20.10.7

Hi @alexcu , thanks for a really well worded question :slight_smile: Unfortunately I don’t have the answer. I asked a minor clarifying point on your x-post here.

Thanks @rmoff and for the suggestions in my cross post! It is quite the strange issue. Is there an alternative means to install the connector instead of using the confluent-hub command?

Yes, you can download the JAR directly from Confluent Hub, or indeed from the Debezium site itself.

But - it sounds like it’s the plugin itself throwing the error, not the Confluent Hub CLI?

Yes it’s the plug-in itself. I did raise an issue with Debezium here which sounds like it could be an installation issue, but oddly the issue only occurs on the M1. I can try and install manually instead of using Confluent Hub, and see if the issue persists.

To confirm, the Debezium MySQL plugin loads perfectly fine.

It loads fine but it looks like Debezium (rather than Kafka Connect) throwing the error in the stack trace that you shared. Let me see if I can summon someone who is more help than me…

[edit] sorry, I skim-read your answer - didn’t realise you said the MySQL one works fine

Yes – it seems the Postgres plugin can’t load. So it does not seem to be a Kafka Connect issue? If you know anyone who can help on the Debezium end, that would be amazing :pray: Thanks @rmoff!

Seems that installing outside confluent-hub loads correctly, using the Dockerfile:

FROM confluentinc/cp-kafka-connect:6.2.0

RUN wget https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/1.6.0.Final/debezium-connector-postgres-1.6.0.Final-plugin.tar.gz &&\
  tar -xf debezium-connector-postgres-1.6.0.Final-plugin.tar.gz &&\
  mv debezium-connector-postgres /usr/share/confluent-hub-components

Now it loads fine:

[2021-07-13 10:50:30,017] INFO Loading plugin from: /usr/share/confluent-hub-components/debezium-connector-postgres (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2021-07-13 10:50:31,090] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/confluent-hub-components/debezium-connector-postgres/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2021-07-13 10:50:31,091] INFO Added plugin 'io.debezium.connector.postgresql.PostgresConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2021-07-13 10:50:31,092] INFO Added plugin 'io.debezium.converters.ByteBufferConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2021-07-13 10:50:31,093] INFO Added plugin 'io.debezium.converters.CloudEventsConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2021-07-13 10:50:31,094] INFO Added plugin 'io.debezium.transforms.outbox.EventRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2021-07-13 10:50:31,094] INFO Added plugin 'io.debezium.transforms.ExtractNewRecordState' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2021-07-13 10:50:31,094] INFO Added plugin 'io.debezium.transforms.ByLogicalTableRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2021-07-13 10:50:31,094] INFO Added plugin 'io.debezium.transforms.tracing.ActivateTracingSpan' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2021-07-13 10:50:38,064] INFO Registered loader: jdk.internal.loader.ClassLoaders$AppClassLoader@9e89d68 (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2021-07-13 10:50:38,072] INFO Added aliases 'PostgresConnector' and 'Postgres' to plugin 'io.debezium.connector.postgresql.PostgresConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)

However, attempting to create the KafkaAdminClient fails:

[2021-07-13 10:50:38,377] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed)
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:540)
	at org.apache.kafka.clients.admin.Admin.create(Admin.java:144)
	at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:50)
	at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:97)
	at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80)
Caused by: org.apache.kafka.common.config.ConfigException: Invalid url in bootstrap.servers: 'broker:29092'
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:59)
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:48)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:494)
	... 4 more

The invalid broker URL is confusing, since it seems to be set up correctly?

Looks like we’re making progress :slight_smile: How about if you just run it as a Docker Compose to make the networking side of things simpler? and then work backwards from there if you need the individual container invocations

Here’s a sample Docker Compose you can slot into: demo-scene/docker-compose.yml at 93b0676a07e7866a13ab2842c43777c221fdfe3f · confluentinc/demo-scene · GitHub

Ha - well, this was originally all in one Docker Compose file (with the Dockefile being the one(s) provided earlier), but I had deconstructed it as a sanity check.

I will re-try from the file you provided and report back.

Okay, I retried with the file provided above, which I slightly modified to use the Postgresql connector instead of MySQL, upgrade to the 6.2.0 images, and to remove others that we do not need.

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.2.0
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka:6.2.0
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 100

  connect:
    image: confluentinc/cp-kafka-connect-base:6.2.0
    container_name: kafka-connect
    depends_on:
      - broker
    ports:
      - 8083:8083
    environment:
      CONNECT_BOOTSTRAP_SERVERS: "broker:29092"
      CONNECT_REST_PORT: 8083
      CONNECT_GROUP_ID: kafka-connect
      CONNECT_CONFIG_STORAGE_TOPIC: _connect-configs
      CONNECT_OFFSET_STORAGE_TOPIC: _connect-offsets
      CONNECT_STATUS_STORAGE_TOPIC: _connect-status
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
      CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081'
      CONNECT_REST_ADVERTISED_HOST_NAME: "kafka-connect"
      CONNECT_LOG4J_APPENDER_STDOUT_LAYOUT_CONVERSIONPATTERN: "[%d] %p %X{connector.context}%m (%c:%L)%n"
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: "1"
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: "1"
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: "1"
      CONNECT_PLUGIN_PATH: /usr/share/java,/usr/share/confluent-hub-components,/data/connect-jars
    volumes:
      - $PWD/data:/data
    command:
      - bash
      - -c
      - |
        echo "Installing Connector"
        confluent-hub install --no-prompt debezium/debezium-connector-postgresql:1.6.0
        #
        echo "Launching Kafka Connect worker"
        /etc/confluent/docker/run &
        #
        sleep infinity

However, still facing the same error:

===> User
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
===> Configuring ...
===> Running preflight checks ... 
===> Check if /var/lib/kafka/data is writable ...
===> Check if Zookeeper is healthy ...
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/share/java/cp-base-new/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/cp-base-new/slf4j-simple-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN No appenders could be found for logger (io.confluent.admin.utils.cli.ZookeeperReadyCommand).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

...

[2021-07-13 12:24:37,822] INFO Loading plugin from: /usr/share/confluent-hub-components/debezium-debezium-connector-postgresql (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:246)
[2021-07-13 12:24:38,546] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed:86)
java.lang.ExceptionInInitializerError
	at io.debezium.connector.postgresql.PostgresConnector.version(PostgresConnector.java:45)
	at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.versionFor(DelegatingClassLoader.java:387)
	at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.versionFor(DelegatingClassLoader.java:392)
	at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:362)
	at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:334)
	at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:268)
	at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:260)
	at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:229)
	at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:206)
	at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
	at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:93)
	at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80)
Caused by: java.lang.NullPointerException: inStream parameter is null
	at java.base/java.util.Objects.requireNonNull(Objects.java:246)
	at java.base/java.util.Properties.load(Properties.java:406)
	at io.debezium.util.IoUtil.loadProperties(IoUtil.java:493)
	at io.debezium.util.IoUtil.loadProperties(IoUtil.java:523)
	at io.debezium.connector.postgresql.Module.<clinit>(Module.java:19)
	... 12 more

Any info?

How about if you install the connector directly, which you found worked earlier?

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.2.0
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka:6.2.0
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 100

  connect:
    image: confluentinc/cp-kafka-connect-base:6.2.0
    container_name: kafka-connect
    depends_on:
      - broker
    ports:
      - 8083:8083
    environment:
      CONNECT_BOOTSTRAP_SERVERS: "broker:29092"
      CONNECT_REST_PORT: 8083
      CONNECT_GROUP_ID: kafka-connect
      CONNECT_CONFIG_STORAGE_TOPIC: _connect-configs
      CONNECT_OFFSET_STORAGE_TOPIC: _connect-offsets
      CONNECT_STATUS_STORAGE_TOPIC: _connect-status
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
      CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081'
      CONNECT_REST_ADVERTISED_HOST_NAME: "kafka-connect"
      CONNECT_LOG4J_APPENDER_STDOUT_LAYOUT_CONVERSIONPATTERN: "[%d] %p %X{connector.context}%m (%c:%L)%n"
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: "1"
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: "1"
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: "1"
      CONNECT_PLUGIN_PATH: /usr/share/java,/usr/share/confluent-hub-components,/data/connect-jars
    volumes:
      - $PWD/data:/data
    command:
      - bash
      - -c
      - |
        echo "Installing Connector"
        wget https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/1.6.0.Final/debezium-connector-postgres-1.6.0.Final-plugin.tar.gz &&\
        tar -xf debezium-connector-postgres-1.6.0.Final-plugin.tar.gz &&\
        mv debezium-connector-postgres /usr/share/confluent-hub-components
        #confluent-hub install --no-prompt debezium/debezium-connector-postgresql:1.6.0
        #
        echo "Launching Kafka Connect worker"
        /etc/confluent/docker/run &
        #
        sleep infinity

Okay, this is getting somewhere now. The plugin loads as expected:

[2021-07-14 11:07:56,723] INFO Loading plugin from: /usr/share/confluent-hub-components/debezium-connector-postgres (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:246)
[2021-07-14 11:07:57,318] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/confluent-hub-components/debezium-connector-postgres/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:269)
[2021-07-14 11:07:57,318] INFO Added plugin 'io.debezium.connector.postgresql.PostgresConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:198)
[2021-07-14 11:07:57,319] INFO Added plugin 'io.debezium.converters.ByteBufferConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:198)
[2021-07-14 11:07:57,319] INFO Added plugin 'io.debezium.converters.CloudEventsConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:198)
[2021-07-14 11:07:57,319] INFO Added plugin 'io.debezium.transforms.outbox.EventRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:198)
[2021-07-14 11:07:57,320] INFO Added plugin 'io.debezium.transforms.ExtractNewRecordState' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:198)
[2021-07-14 11:07:57,320] INFO Added plugin 'io.debezium.transforms.ByLogicalTableRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:198)
[2021-07-14 11:07:57,320] INFO Added plugin 'io.debezium.transforms.tracing.ActivateTracingSpan' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:198)
[2021-07-14 11:08:03,142] INFO Registered loader: jdk.internal.loader.ClassLoaders$AppClassLoader@9e89d68 (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:269)
[2021-07-14 11:08:03,147] INFO Added aliases 'PostgresConnector' and 'Postgres' to plugin 'io.debezium.connector.postgresql.PostgresConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:427)

However, I’m now seeing this error come up causing the connector to stop:

[2021-07-14 11:08:07,482] INFO Adding admin resources to main listener (org.apache.kafka.connect.runtime.rest.RestServer:241)
[2021-07-14 11:08:07,817] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed:86)
java.lang.NoClassDefFoundError: Could not initialize class org.eclipse.jetty.http.MimeTypes$Type
	at org.eclipse.jetty.http.MimeTypes.<clinit>(MimeTypes.java:175)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:895)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.apache.kafka.connect.runtime.rest.RestServer.initializeResources(RestServer.java:305)
	at org.apache.kafka.connect.runtime.Connect.start(Connect.java:55)
	at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:140)
	at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80)
[2021-07-14 11:08:07,841] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:67)
[2021-07-14 11:08:07,863] INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer:327)
[2021-07-14 11:08:08,017] INFO Stopped http_8083@4cb3e76c{HTTP/1.1, (http/1.1)}{0.0.0.0:8083} (org.eclipse.jetty.server.AbstractConnector:381)
[2021-07-14 11:08:08,223] INFO REST server stopped (org.apache.kafka.connect.runtime.rest.RestServer:344)

This eventually causes the connector to stop:

[2021-07-14 11:08:13,139] INFO Kafka Connect stopped (org.apache.kafka.connect.runtime.Connect:72)

Did you see this issue when you ran it?

I placed the full logs here: Kafka Connect Postgresql Issue · GitHub

Thanks for your help @rmoff, certainly closer to isolating the issue now!

Based on this Slack conversation with Sandeep in the Confluent Community workspace, the issue appears to be:

The above error happens because of version mismatch when you install Java packages :thinking_face:

I’m out of ideas here. If it’s not working even from direct install (per your most recent Docker Compose here) then I’d maybe try the Debezium folk directly and see if they can help : debezium/user - Gitter

From my understanding, the connector seems to install fine with the direct install approach. However, when the connector proceeds to add admin resources, the org.eclipse.jetty.http.MimeTypes class isn’t found:

I am not familiar enough with Kafka to understand what could be going wrong here, but I have a feeling that this may no longer be an issue related to the Debezium plugin? I’ll reach out to the Debezium community to look. Alternatively, I’ll also see if the MySQL connector has the same issue…

Running this with the MySQL connector has interesting behaviour. With a manual install, I see the same error:

[2021-07-17 01:11:16,496] INFO Kafka version: 6.2.0-ccs (org.apache.kafka.common.utils.AppInfoParser:119)
[2021-07-17 01:11:16,496] INFO Kafka commitId: 1a5755cf9401c84f (org.apache.kafka.common.utils.AppInfoParser:120)
[2021-07-17 01:11:16,497] INFO Kafka startTimeMs: 1626484276496 (org.apache.kafka.common.utils.AppInfoParser:121)
[2021-07-17 01:11:16,788] INFO Adding admin resources to main listener (org.apache.kafka.connect.runtime.rest.RestServer:241)
[2021-07-17 01:11:17,170] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed:86)
java.lang.NoClassDefFoundError: Could not initialize class org.eclipse.jetty.http.MimeTypes$Type
	at org.eclipse.jetty.http.MimeTypes.<clinit>(MimeTypes.java:175)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:895)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.apache.kafka.connect.runtime.rest.RestServer.initializeResources(RestServer.java:305)
	at org.apache.kafka.connect.runtime.Connect.start(Connect.java:55)
	at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:140)
	at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80)
[2021-07-17 01:11:17,190] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:67)

(Full log for MySQL via manual install here: Kafka Connect Postgresql Issue · GitHub)

However, if I install via confluent-hub (which works fine, unlike the Postgres connector), I see the above error in addition to the following:

[2021-07-17 01:11:21,941] INFO Started KafkaBasedLog for topic _connect-status (org.apache.kafka.connect.util.KafkaBasedLog:204)
[2021-07-17 01:11:21,950] INFO Starting KafkaConfigBackingStore (org.apache.kafka.connect.storage.KafkaConfigBackingStore:275)
[2021-07-17 01:11:21,953] INFO Starting KafkaBasedLog with topic _connect-configs (org.apache.kafka.connect.util.KafkaBasedLog:162)
[2021-07-17 01:11:22,247] INFO [Worker clientId=connect-1, groupId=kafka-connect] Herder stopped (org.apache.kafka.connect.runtime.distributed.DistributedHerder:738)
[2021-07-17 01:11:22,248] ERROR [Worker clientId=connect-1, groupId=kafka-connect] Uncaught exception in herder work thread, exiting:  (org.apache.kafka.connect.runtime.distributed.DistributedHerder:324)
org.apache.kafka.connect.errors.ConnectException: Interrupted while attempting to create/find topic(s) '_connect-configs'
	at org.apache.kafka.connect.util.TopicAdmin.createOrFindTopics(TopicAdmin.java:430)
	at org.apache.kafka.connect.util.TopicAdmin.createTopics(TopicAdmin.java:332)
	at org.apache.kafka.connect.storage.KafkaConfigBackingStore.lambda$createKafkaBasedLog$0(KafkaConfigBackingStore.java:518)
	at org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:166)
	at org.apache.kafka.connect.storage.KafkaConfigBackingStore.start(KafkaConfigBackingStore.java:278)
	at org.apache.kafka.connect.runtime.AbstractHerder.startServices(AbstractHerder.java:130)
	at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:310)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.InterruptedException
	at java.base/java.lang.Object.wait(Native Method)
	at java.base/java.lang.Object.wait(Object.java:328)
	at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:92)
	at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
	at org.apache.kafka.connect.util.TopicAdmin.createOrFindTopics(TopicAdmin.java:387)
	... 11 more
[2021-07-17 01:11:22,248] INFO Kafka Connect stopped (org.apache.kafka.connect.runtime.Connect:72)

(Full log for MySQL via confluent-hub here: Kafka Connect Postgresql Issue · GitHub)

Okay, I tried with a non-Debezium plugin: this time the [Confluent JDBC] plugin installed via confluent-hub. And, I am still seeing the same error:

[2021-07-17 01:27:29,960] INFO Adding admin resources to main listener (org.apache.kafka.connect.runtime.rest.RestServer:241)
[2021-07-17 01:27:30,558] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed:86)
java.lang.ExceptionInInitializerError
	at org.eclipse.jetty.http.MimeTypes.<clinit>(MimeTypes.java:175)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:895)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.apache.kafka.connect.runtime.rest.RestServer.initializeResources(RestServer.java:305)
	at org.apache.kafka.connect.runtime.Connect.start(Connect.java:55)
	at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:140)
	at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80)
Caused by: java.nio.charset.IllegalCharsetNameException: l;charset=iso-8859-1
	at java.base/java.nio.charset.Charset.checkName(Charset.java:308)
	at java.base/java.nio.charset.Charset.lookup2(Charset.java:482)
	at java.base/java.nio.charset.Charset.lookup(Charset.java:462)
	at java.base/java.nio.charset.Charset.forName(Charset.java:526)
	at org.eclipse.jetty.http.MimeTypes$Type.<init>(MimeTypes.java:107)
	at org.eclipse.jetty.http.MimeTypes$Type.<clinit>(MimeTypes.java:67)
	... 8 more
[2021-07-17 01:27:30,592] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:67)
[2021-07-17 01:27:30,603] INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer:327)

(Logs and docker-compose.yml)

So, it does not seem to be an issue now on the Debezium end…