Installing Custom Connector

Kafka Connect Self-Managed Connectors
1

Hi,
I have a java maven project where I’m using an assembly plugin to create a directory containing config, jar file, and all dependencies. I’m mounting this to a directory like /usr/local/share/myconnector and running connect via docker with plugin path set to /usr/local/share. I can see in the logs that it ‘sees’ myconnector, but when I check curl http://localhost:8083/connector-plugins |jqits not there. So when I try to do something like:
curl -s -X PUT -H 'Content-Type: application/json' --data @SoupConnector.json http://localhost:8083/connectors/DansConnector/config

I get the obvious exception that “Failed to find any class that implements Connector and which name matches… ”.

I’m unclear on exactly what needs to be in this directory that I mount - obviously all the dependency jars and my compiled connector jar, all just in the root dir? Do I need config files? Are there any loaders that I can increase log levels to show me what it’s finding in the directory and why it’s not loading the connector jar? I’m at a loss, everything looks fine when I exec in but I’m new to deploying a custom connector and a lot on the web is getting a bit old.

Let me know what else I can check.
Thanks in advance,
Dan C.

2

FYI I’m running confluentinc/cp-kafka-connect-base:7.4.0.arm64 on a Mac M1 with Java 17.

I’ve found that if I change my java compiler to 1.8 src/target, then the connector does show up…

Literally this is the only thing I change in my pom - and it works:

   <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>${maven-compiler.version}</version>
                <inherited>true</inherited>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>

Adding that source/target config… as opposed to my default java 17 setup.

This doesn’t seem right though… any thoughts?

3

I can’t find any examples of connectors that have been written using features from java 17. And I’m unable to find any apache / confluent documentation saying that connectors must be written in something < 17.

Weird. Would like confirmation if anyone can provide it - alternatively a solution to packaging up a connector with java 17 would be great. I haven’t tried uber jar yet.

4

Use the Maven plugin kafka-connect-maven-plugin – Dependency Information

5

Thanks @OneCricketeer - package plugin works well unfortunately still not able to get it to show up! Pretty frustrated at this point… no logs etc. You wouldn’t happen to have a ‘hello world’ example of a java 17 connector mvn project would you? Can’t figure it out - no log info of note - running latest docker/connect 7.4.0-arm64/java libraries/etc…

I’ve looked at the confluent connectors in github and as far as I can tell they still use assembly plugin and java 1.8 configs.

Cheers

6

Is there a specific API of Java 17 you’re trying to use?

In practice, compiling to Java 8 would provide the best compatibility with Connect runtime. Plus, last I checked, the Confluent containers don’t run Java 17

7

Wow… yea this is the 7.4.0 container:
image

But the ‘recommended java version’ is 17 for Confluent 7.4 according to this.

I guess that doesn’t extend to their containers.

Probably not an easy fix as it seems they have code depending on class loader implementations… I see this in connect debug logs on finding my connector, not sure if there’s a way around it:

image
image1121×262 53.4 KB

8

Found it - there’s an issue request in confluent common-docker repo to update base images to java 17. Started a while ago with response saying java 17 wasn’t the recommended version nor on the roadmap… but later people point out with Kafka 3 and confluent 7 - java 17 is now being documented as the recommended java version for confluent and kafka, yet they have not updated their base images.

Guess I’ll watch this issue. Our libraries have too many new jdk-17 specific implementations that choke on java 8/11 target compiles.

9

You’re welcome to try a different image, such as mine - GitHub - OneCricketeer/apache-kafka-connect-docker: [KAFKA-9774] Un-official Docker Image for Apache Kafka Connect

10

Thanks @OneCricketeer - probably the best option for now while we wait for the above ticket. Nice work.

11

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.