JDBC sink connector is not displayed under the available connectors

Kafka Connect Managed Connectors
1

Hi Team,

Command - confluent local services connect connector list doesn’t list JDBCSink and Source connectors.

I tried installing them manually using confluent-hub install confluentinc/kafka-connect-jdbc:10.1.0 . Then also, it was listed out.

Can you please help. Plugin path was added automatically while running the previous command.

I only see the below items in the available connectors list,

The local commands are intended for a single-node development environment only,
NOT for production usage. https://docs.confluent.io/current/cli/index.html

Bundled Connectors:

  • file-sink*
  • file-source*
  • replicator*
2

Did you restart Kafka Connect after installing the plugin?

After restarting Kafka Connect, what’s the output of curl -s localhost:8083/connector-plugins ?

3

I restarted the entire service using the commands, confluent local services stop and confluent local services start.

Below is the output curl -s localhost:8083/connector-plugins

[
  {
    "class": "io.confluent.connect.replicator.ReplicatorSourceConnector",
    "type": "source",
    "version": "6.1.1"
  },
  {
    "class": "org.apache.kafka.connect.file.FileStreamSinkConnector",
    "type": "sink",
    "version": "6.1.1-ce"
  },
  {
    "class": "org.apache.kafka.connect.file.FileStreamSourceConnector",
    "type": "source",
    "version": "6.1.1-ce"
  },
  {
    "class": "org.apache.kafka.connect.mirror.MirrorCheckpointConnector",
    "type": "source",
    "version": "1"
  },
  {
    "class": "org.apache.kafka.connect.mirror.MirrorHeartbeatConnector",
    "type": "source",
    "version": "1"
  },
  {
    "class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
    "type": "source",
    "version": "1"
  }
]
4

Can you paste the full output of your terminal when you run confluent-hub install confluentinc/kafka-connect-jdbc:10.1.0 please?

5

I reinstalled it. here is the raw output:

vigneshmanivasagan@BLRJPT6950L ~ % confluent-hub install confluentinc/kafka-connect-jdbc:10.1.0
The component can be installed in any of the following Confluent Platform installations: 
  1. /Users/vigneshmanivasagan/Documents/confluent-6.1.1 (based on $CONFLUENT_HOME) 
  2. /Users/vigneshmanivasagan/Documents/confluent-6.1.1 (where this tool is installed) 
Choose one of these to continue the installation (1-2): 1
Do you want to install this into /Users/vigneshmanivasagan/Documents/confluent-6.1.1/share/confluent-hub-components? (yN) y

 
Component's license: 
Confluent Community License 
https://www.confluent.io/confluent-community-license 
I agree to the software license agreement (yN) y

Downloading component Kafka Connect JDBC 10.1.0, provided by Confluent, Inc. from Confluent Hub and installing into /Users/vigneshmanivasagan/Documents/confluent-6.1.1/share/confluent-hub-components 
Do you want to uninstall existing version 10.1.0? (yN) y

Detected Worker's configs: 
  1. Standard: /Users/vigneshmanivasagan/Documents/confluent-6.1.1/etc/kafka/connect-distributed.properties 
  2. Standard: /Users/vigneshmanivasagan/Documents/confluent-6.1.1/etc/kafka/connect-standalone.properties 
  3. Standard: /Users/vigneshmanivasagan/Documents/confluent-6.1.1/etc/schema-registry/connect-avro-distributed.properties 
  4. Standard: /Users/vigneshmanivasagan/Documents/confluent-6.1.1/etc/schema-registry/connect-avro-standalone.properties 
  5. Based on CONFLUENT_CURRENT: /var/folders/bs/rp8j3n8d05388z1rn7wclhrm0000gq/T/confluent.524407/connect/connect.properties 
  6. Used by Connect process with PID 11785: /var/folders/bs/rp8j3n8d05388z1rn7wclhrm0000gq/T/confluent.524407/connect/connect.properties 
Do you want to update all detected configs? (yN) y

Adding installation directory to plugin path in the following files: 
  /Users/vigneshmanivasagan/Documents/confluent-6.1.1/etc/kafka/connect-distributed.properties 
  /Users/vigneshmanivasagan/Documents/confluent-6.1.1/etc/kafka/connect-standalone.properties 
  /Users/vigneshmanivasagan/Documents/confluent-6.1.1/etc/schema-registry/connect-avro-distributed.properties 
  /Users/vigneshmanivasagan/Documents/confluent-6.1.1/etc/schema-registry/connect-avro-standalone.properties 
  /var/folders/bs/rp8j3n8d05388z1rn7wclhrm0000gq/T/confluent.524407/connect/connect.properties 
  /var/folders/bs/rp8j3n8d05388z1rn7wclhrm0000gq/T/confluent.524407/connect/connect.properties 
 
Completed 
vigneshmanivasagan@BLRJPT6950L ~ %
6

What’s the output of these two commands?

ls -l /Users/vigneshmanivasagan/Documents/confluent-6.1.1/share/confluent-hub-components

and

confluent local services connect log|grep -i plugin.path
7

Also I’ve just realised a mistake in your command.

This lists the available connectors that have been configured:

confluent local services connect connector list

This lists the available plugins that have been installed and can be used to create a connector:

confluent local services connect plugin list
8

Output:
drwxr-xr-x 8 vigneshmanivasagan staff 256 Mar 31 19:33 confluentinc-kafka-connect-datagen
drwxr-xr-x 7 vigneshmanivasagan staff 224 Mar 31 21:41 confluentinc-kafka-connect-jdbc

9

Output:

vigneshmanivasagan@BLRJPT6950L ~ % confluent local services connect log|grep -i plugin.path

The local commands are intended for a single-node development environment only,

NOT for production usage. https://docs.confluent.io/current/cli/index.html

plugin.path = [/Users/vigneshmanivasagan/Documents/confluent-6.1.1/share/java]

[2021-03-31 19:21:37,113] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:369)

[2021-03-31 19:21:37,913] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:369)

[2021-03-31 19:21:37,958] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:369)

[2021-03-31 19:21:38,333] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:369)

[2021-03-31 19:21:38,384] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:369)

[2021-03-31 19:21:38,421] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:369)

[2021-03-31 19:21:38,478] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:369)

[2021-03-31 19:21:38,559] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:369)

[2021-03-31 19:21:38,688] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig:369)

[2021-03-31 19:21:38,768] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:369)

[2021-03-31 19:21:38,890] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig:369)

[2021-03-31 19:21:38,901] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:369)

[2021-03-31 19:21:39,006] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig:369)

[2021-03-31 19:21:39,026] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:369)
10

3rd and 4th suggested command’s output is almost same.

11

I’ve solved this issue using by

  1. using this command confluent local services connect log|grep -i plugin.path i found where the displayed jars are present.
  2. and then I made sure my jdbc jars and other jars like postgre jar are present in that path.
  • ideally when we install jars, confluent itself alters the plugin path for us, we need to ensure the updated path is correct.
12

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.