JDBC sink connector is not displayed under the available connectors

vmanivasagan · 31 March 2021 14:39

Hi Team,

Command - confluent local services connect connector list doesn’t list JDBCSink and Source connectors.

I tried installing them manually using confluent-hub install confluentinc/kafka-connect-jdbc:10.1.0 . Then also, it was listed out.

Can you please help. Plugin path was added automatically while running the previous command.

I only see the below items in the available connectors list,

The local commands are intended for a single-node development environment only,
NOT for production usage. https://docs.confluent.io/current/cli/index.html

Bundled Connectors:

file-sink*
file-source*
replicator*

rmoff · 31 March 2021 14:40

Did you restart Kafka Connect after installing the plugin?

After restarting Kafka Connect, what’s the output of curl -s localhost:8083/connector-plugins ?

vmanivasagan · 31 March 2021 16:05

I restarted the entire service using the commands, confluent local services stop and confluent local services start.

Below is the output curl -s localhost:8083/connector-plugins

[
  {
    "class": "io.confluent.connect.replicator.ReplicatorSourceConnector",
    "type": "source",
    "version": "6.1.1"
  },
  {
    "class": "org.apache.kafka.connect.file.FileStreamSinkConnector",
    "type": "sink",
    "version": "6.1.1-ce"
  },
  {
    "class": "org.apache.kafka.connect.file.FileStreamSourceConnector",
    "type": "source",
    "version": "6.1.1-ce"
  },
  {
    "class": "org.apache.kafka.connect.mirror.MirrorCheckpointConnector",
    "type": "source",
    "version": "1"
  },
  {
    "class": "org.apache.kafka.connect.mirror.MirrorHeartbeatConnector",
    "type": "source",
    "version": "1"
  },
  {
    "class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
    "type": "source",
    "version": "1"
  }
]

rmoff · 31 March 2021 16:09

Can you paste the full output of your terminal when you run confluent-hub install confluentinc/kafka-connect-jdbc:10.1.0 please?

vmanivasagan · 31 March 2021 17:06

I reinstalled it. here is the raw output:

vigneshmanivasagan@BLRJPT6950L ~ % confluent-hub install confluentinc/kafka-connect-jdbc:10.1.0
The component can be installed in any of the following Confluent Platform installations: 
  1. /Users/vigneshmanivasagan/Documents/confluent-6.1.1 (based on $CONFLUENT_HOME) 
  2. /Users/vigneshmanivasagan/Documents/confluent-6.1.1 (where this tool is installed) 
Choose one of these to continue the installation (1-2): 1
Do you want to install this into /Users/vigneshmanivasagan/Documents/confluent-6.1.1/share/confluent-hub-components? (yN) y

 
Component's license: 
Confluent Community License 
https://www.confluent.io/confluent-community-license 
I agree to the software license agreement (yN) y

Downloading component Kafka Connect JDBC 10.1.0, provided by Confluent, Inc. from Confluent Hub and installing into /Users/vigneshmanivasagan/Documents/confluent-6.1.1/share/confluent-hub-components 
Do you want to uninstall existing version 10.1.0? (yN) y

Detected Worker's configs: 
  1. Standard: /Users/vigneshmanivasagan/Documents/confluent-6.1.1/etc/kafka/connect-distributed.properties 
  2. Standard: /Users/vigneshmanivasagan/Documents/confluent-6.1.1/etc/kafka/connect-standalone.properties 
  3. Standard: /Users/vigneshmanivasagan/Documents/confluent-6.1.1/etc/schema-registry/connect-avro-distributed.properties 
  4. Standard: /Users/vigneshmanivasagan/Documents/confluent-6.1.1/etc/schema-registry/connect-avro-standalone.properties 
  5. Based on CONFLUENT_CURRENT: /var/folders/bs/rp8j3n8d05388z1rn7wclhrm0000gq/T/confluent.524407/connect/connect.properties 
  6. Used by Connect process with PID 11785: /var/folders/bs/rp8j3n8d05388z1rn7wclhrm0000gq/T/confluent.524407/connect/connect.properties 
Do you want to update all detected configs? (yN) y

Adding installation directory to plugin path in the following files: 
  /Users/vigneshmanivasagan/Documents/confluent-6.1.1/etc/kafka/connect-distributed.properties 
  /Users/vigneshmanivasagan/Documents/confluent-6.1.1/etc/kafka/connect-standalone.properties 
  /Users/vigneshmanivasagan/Documents/confluent-6.1.1/etc/schema-registry/connect-avro-distributed.properties 
  /Users/vigneshmanivasagan/Documents/confluent-6.1.1/etc/schema-registry/connect-avro-standalone.properties 
  /var/folders/bs/rp8j3n8d05388z1rn7wclhrm0000gq/T/confluent.524407/connect/connect.properties 
  /var/folders/bs/rp8j3n8d05388z1rn7wclhrm0000gq/T/confluent.524407/connect/connect.properties 
 
Completed 
vigneshmanivasagan@BLRJPT6950L ~ %

rmoff · 3 April 2021 16:13

What’s the output of these two commands?

ls -l /Users/vigneshmanivasagan/Documents/confluent-6.1.1/share/confluent-hub-components

and

confluent local services connect log|grep -i plugin.path

rmoff · 3 April 2021 16:13

Also I’ve just realised a mistake in your command.

This lists the available connectors that have been configured:

confluent local services connect connector list

This lists the available plugins that have been installed and can be used to create a connector:

confluent local services connect plugin list

vmanivasagan · 31 March 2021 17:58

Output:
drwxr-xr-x 8 vigneshmanivasagan staff 256 Mar 31 19:33 confluentinc-kafka-connect-datagen
drwxr-xr-x 7 vigneshmanivasagan staff 224 Mar 31 21:41 confluentinc-kafka-connect-jdbc

vmanivasagan · 31 March 2021 18:33

Output:

vigneshmanivasagan@BLRJPT6950L ~ % confluent local services connect log|grep -i plugin.path

The local commands are intended for a single-node development environment only,

NOT for production usage. https://docs.confluent.io/current/cli/index.html

plugin.path = [/Users/vigneshmanivasagan/Documents/confluent-6.1.1/share/java]

[2021-03-31 19:21:37,113] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:369)

[2021-03-31 19:21:37,913] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:369)

[2021-03-31 19:21:37,958] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:369)

[2021-03-31 19:21:38,333] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:369)

[2021-03-31 19:21:38,384] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:369)

[2021-03-31 19:21:38,421] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:369)

[2021-03-31 19:21:38,478] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:369)

[2021-03-31 19:21:38,559] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:369)

[2021-03-31 19:21:38,688] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig:369)

[2021-03-31 19:21:38,768] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:369)

[2021-03-31 19:21:38,890] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig:369)

[2021-03-31 19:21:38,901] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:369)

[2021-03-31 19:21:39,006] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig:369)

[2021-03-31 19:21:39,026] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:369)

vmanivasagan · 31 March 2021 17:59

3rd and 4th suggested command’s output is almost same.

vmanivasagan · 3 April 2021 16:13

I’ve solved this issue using by

using this command confluent local services connect log|grep -i plugin.path i found where the displayed jars are present.
and then I made sure my jdbc jars and other jars like postgre jar are present in that path.

ideally when we install jars, confluent itself alters the plugin path for us, we need to ensure the updated path is correct.

system · 10 April 2021 16:13

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.