The mystery of CUB_CLASSPATH in the land of Kafka Connect

Kafka Connect
1

Hi Kafkafarians,

I am maintaining a Kafka Connect cluster at work that is based on the confluentinc/cp-kafka-connect-base:5.4.1 Docker image.

We have written a number of plugins, each packaged as a zip file with a lib directory containing all that plugins jar files. None of the plugins have manifest.json files (is the format of these files documented anywhere?). All these plugins are custom ConfigProvider implementations.

For some reason, not only are they being added to a directory on CONNECT_PLUGIN_PATH, but they are also added to a directory on CUB_CLASSPATH.

I asked the original engineer who had built these customized Dockerfiles about this and they said they have done that because otherwise Kafka Connect wouldn’t recognize them.

I am in the process of replacing some COPY/ADD commands with RUN confluent-hub install alternatives but I noticed unlike libraries on Confluent Hub that can easily be installed by running something like:

RUN confluent-hub install --no-prompt confluentinc/kafka-connect-jdbc:10.0.0

If I install our custom ConfigProvider plugins using RUN confluent-hub install --no-prompt they won’t get loaded and become available for use in connector definitions.

  • Is this a known issue or expected behavior?
  • Has it been fixed since kafka-connect-base:5.4.1?

Thanks in advance.

2

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.