Confluent for Kubernetes - unable to setup plugins

binnsr · 24 January 2025 16:18

Hi - pardon my ignorance if this is a dumb question as I’m completely new to connect.
I’m trying to deploy connect using confluent for kubernetes and have developed a helm chart that can get a pod up and running, but when I add the connect.spec.build parameter, it still starts up, but I get an error in the pod log:

ERROR Unable to update dynamic config (org.apache.kafka.connect.runtime.DynamicWorkerConfigManager:74)

java.nio.file.NoSuchFileException: /mnt/config/shared/dynamic-worker-config.properties

My manifest looks like this:

---
# Source: confluent-chart/templates/connect/connect.yaml
apiVersion: platform.confluent.io/v1beta1
kind: Connect
metadata:
  name: confluent-connect
  namespace: dev
spec:
  image:
    application: confluentinc/cp-server-connect:7.8.0
    init: confluentinc/confluent-init-container:2.10.0
  replicas: 1
  dependencies:
    kafka:
      bootstrapEndpoint: REDACTED
      authentication:
        type: mtls
      tls:
        enabled: true
        secretRef: REDACTED
  oneReplicaPerNode: true
  internalTopicReplicationFactor: 3
  connectorTLSCerts:
    - secretRef: REDACTED
  mountedSecrets:
    - secretRef: REDACTED
  keyConverterType: "org.apache.kafka.connect.storage.StringConverter"
  valueConverterType: "org.apache.kafka.connect.json.JsonConverter"
  build:
    type: onDemand
    onDemand:
      plugins:
        confluentHub:
          - name: kafka-connect-datagen ## DO NOT USE THIS ONE IN PRODUCTION
            owner: confluentinc
            version: latest
      storageLimit: 4G
  podTemplate:
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: confluent
              operator: "In"
              values:
                - "true"
    topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: type
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            .confluent: "true"

What am I missing??

I’ve not been able to find any reference (either in the rather disjointed confluent pages or in kubectl explain to help me get a worker config file generated here, so any assistance would be appreciated…

dtroiano · 24 January 2025 19:33

Based on some internal discussion I’m seeing about this particular error, it looks to be noise that you can ignore. It will likely be cleaned up in the next cp-server-connect release.

We always like to receive specific feedback so that we can improve. Either here, or as a DM to me. If your feedback is on a certain developer.confluent.io page you can use the blue Feedback button on the right of the page:

For a docs.confluent.io page scroll to the bottom where you’ll find a Give us feedback button:

binnsr · 27 January 2025 16:33

Thanks for that – I’ll ignore that error. I managed to get past the build section blowing up, but am still unable to get a connector running, even though the ‘connect’ object starts successfully.
The current error is

  Warning  Warning  113s (x18 over 4m43s)  connector  connect Rest API request failed: Failed to find any class that implements Connector and which name matches io.confluent.kafka.connect.datagen.DatagenConnector, available connectors are: PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorCheckpointConnector, name='org.apache.kafka.connect.mirror.MirrorCheckpointConnector', version='7.8.0-ce', encodedVersion=7.8.0-ce, type=source, typeName='source', location='file:/usr/share/java/kafka/'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorCheckpointConnector, name='org.apache.kafka.connect.mirror.MirrorCheckpointConnector', version='7.8.0-ce', encodedVersion=7.8.0-ce, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorHeartbeatConnector, name='org.apache.kafka.connect.mirror.MirrorHeartbeatConnector', version='7.8.0-ce', encodedVersion=7.8.0-ce, type=source, typeName='source', location='file:/usr/share/java/kafka/'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorHeartbeatConnector, name='org.apache.kafka.connect.mirror.MirrorHeartbeatConnector', version='7.8.0-ce', encodedVersion=7.8.0-ce, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorSourceConnector, name='org.apache.kafka.connect.mirror.MirrorSourceConnector', version='7.8.0-ce', encodedVersion=7.8.0-ce, type=source, typeName='source', location='file:/usr/share/java/kafka/'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorSourceConnector, name='org.apache.kafka.connect.mirror.MirrorSourceConnector', version='7.8.0-ce', encodedVersion=7.8.0-ce, type=source, typeName='source', location='classpath'} (500)

In the kubectl describe for the connect resource, I see the following connector plugins defined:

  Connector Plugins:
    Class:    org.apache.kafka.connect.mirror.MirrorCheckpointConnector
    Type:     source
    Version:  7.8.0-ce
    Class:    org.apache.kafka.connect.mirror.MirrorHeartbeatConnector
    Type:     source
    Version:  7.8.0-ce
    Class:    org.apache.kafka.connect.mirror.MirrorSourceConnector
    Type:     source
    Version:  7.8.0-ce

In my manifest, the .spec.build section specifies the datagen plugin to be pulled and installed…

spec:
  image:
    application: confluentinc/cp-server-connect:latest
    init: confluentinc/confluent-init-container:2.10.0
  replicas: 1
  dependencies:
    kafka:
      bootstrapEndpoint: REDACTED
      authentication:
        type: mtls
      tls:
        enabled: true
        secretRef: "REDACTED"
  oneReplicaPerNode: true
  internalTopicReplicationFactor: 3
  keyConverterType: "org.apache.kafka.connect.storage.StringConverter"
  valueConverterType: "org.apache.kafka.connect.json.JsonConverter"
  mountedSecrets:
    - secretRef: REDACTED
  connectorTLSCerts:
    - secretRef: REDACTED
  build:
    type: onDemand
    onDemand:
      plugins:
        confluentHub:
          - name: kafka-connect-datagen
            owner: confluentinc
            version: 0.5.2
      storageLimit: 4G

edit: here’s the log for my init container:

2025-01-27T17:15:21.429087334Z ===> Installing connector plugins
2025-01-27T17:15:22.721129295Z Running in a "--no-prompt" mode 
2025-01-27T17:15:27.502686442Z Implicit acceptance of the license below:  
2025-01-27T17:15:27.502749700Z Apache License 2.0 
2025-01-27T17:15:27.502753326Z https://www.apache.org/licenses/LICENSE-2.0 
2025-01-27T17:15:27.502882145Z Downloading component Kafka Connect Datagen 0.5.2, provided by Confluent, Inc. from Confluent Hub and installing into /mnt/plugins 
2025-01-27T17:15:29.606694726Z Adding installation directory to plugin path in the following files: 
2025-01-27T17:15:29.607200242Z   /dev/null 
2025-01-27T17:15:29.713130027Z  
2025-01-27T17:15:29.713157490Z Completed 
2025-01-27T17:15:29.793236679Z + rm /mnt/config/install_plugins.sh /mnt/config/plugins.json
2025-01-27T17:15:29.795043720Z + '[' -n connect ']'
2025-01-27T17:15:29.795072914Z load script for CP type: connect
2025-01-27T17:15:29.795089566Z + echo 'load script for CP type: connect'
2025-01-27T17:15:29.795117492Z + cp -r /opt/confluentinc/connect/ /mnt/config/
2025-01-27T17:15:29.798287358Z + echo 'copy fips jvm config security files'
2025-01-27T17:15:29.798314222Z + cp -r /opt/confluentinc/fips/ /mnt/config/
2025-01-27T17:15:29.798319626Z copy fips jvm config security files
2025-01-27T17:15:29.800920678Z + '[' connect = kafka ']'
2025-01-27T17:16:08.485871154Z stream closed EOF for dev/confluent-connect-0 (config-init-container)

Looks like it’s installing the connector properly from that…

dtroiano · 27 January 2025 17:26

Do you see anything in the connect pod logs around where it scans for plugins? You should see logging like:

[2025-01-27 17:14:02,511] INFO Scanning for plugin classes. This might take a moment

...

[2025-01-27 17:15:42,245] INFO Added plugin 'io.confluent.kafka.connect.datagen.DatagenConnector':PluginDesc{klass=class io.confluent.kafka.connect.datagen.DatagenConnector, name='io.confluent.kafka.connect.datagen.DatagenConnector', version='null', encodedVersion=null, type=source, typeName='source', location='file:/mnt/plugins/confluentinc-kafka-connect-datagen/'} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:111)

binnsr · 27 January 2025 17:40

Nope - I get the Scanning message, but not the second…

$ kubectl logs -n dev confluent-connect-0 | grep -E 'Scanning for plugin classes|io.confluent.kafka.connect.datagen.DatagenConnector'
Defaulted container "confluent-connect" out of: confluent-connect, config-init-container (init)
[2025-01-27 17:15:32,510] INFO Scanning for plugin classes. This might take a moment ... (org.apache.kafka.connect.cli.AbstractConnectCli:127)
org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches io.confluent.kafka.connect.datagen.DatagenConnector, available connectors are: PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorCheckpointConnector, name='org.apache.kafka.connect.mirror.MirrorCheckpointConnector', version='7.8.0-ce', encodedVersion=7.8.0-ce, type=source, typeName='source', location='file:/usr/share/java/kafka/'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorCheckpointConnector, name='org.apache.kafka.connect.mirror.MirrorCheckpointConnector', version='7.8.0-ce', encodedVersion=7.8.0-ce, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorHeartbeatConnector, name='org.apache.kafka.connect.mirror.MirrorHeartbeatConnector', version='7.8.0-ce', encodedVersion=7.8.0-ce, type=source, typeName='source', location='file:/usr/share/java/kafka/'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorHeartbeatConnector, name='org.apache.kafka.connect.mirror.MirrorHeartbeatConnector', version='7.8.0-ce', encodedVersion=7.8.0-ce, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorSourceConnector, name='org.apache.kafka.connect.mirror.MirrorSourceConnector', version='7.8.0-ce', encodedVersion=7.8.0-ce, type=source, typeName='source', location='file:/usr/share/java/kafka/'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorSourceConnector, name='org.apache.kafka.connect.mirror.MirrorSourceConnector', version='7.8.0-ce', encodedVersion=7.8.0-ce, type=source, typeName='source', location='classpath'}

I do see this WARN line between the plugin scan lines and the ‘Added plugin’ lines:

file:/usr/share/java/acl/	io.confluent.connect.avro.AvroConverter	converter	undefined
file:/usr/share/java/confluent-control-center/	io.confluent.connect.avro.AvroConverter	converter	undefined
file:/usr/share/java/confluent-security/	io.confluent.connect.avro.AvroConverter	converter	undefined
file:/usr/share/java/kafka-serde-tools/	io.confluent.connect.avro.AvroConverter	converter	undefined
classpath	io.confluent.connect.avro.AvroConverter	converter	undefined
file:/usr/share/java/acl/	io.confluent.connect.json.JsonSchemaConverter	converter	undefined
file:/usr/share/java/confluent-control-center/	io.confluent.connect.json.JsonSchemaConverter	converter	undefined
file:/usr/share/java/confluent-security/	io.confluent.connect.json.JsonSchemaConverter	converter	undefined
file:/usr/share/java/kafka-serde-tools/	io.confluent.connect.json.JsonSchemaConverter	converter	undefined
classpath	io.confluent.connect.json.JsonSchemaConverter	converter	undefined
file:/usr/share/java/acl/	io.confluent.connect.protobuf.ProtobufConverter	converter	undefined
file:/usr/share/java/confluent-control-center/	io.confluent.connect.protobuf.ProtobufConverter	converter	undefined
file:/usr/share/java/confluent-security/	io.confluent.connect.protobuf.ProtobufConverter	converter	undefined
file:/usr/share/java/kafka-serde-tools/	io.confluent.connect.protobuf.ProtobufConverter	converter	undefined
classpath	io.confluent.connect.protobuf.ProtobufConverter	converter	undefined
file:/usr/share/java/kafka/	io.confluent.connect.rest.datapreview.extension.util.PreviewRecordTransformer	transformation	undefined
classpath	io.confluent.connect.rest.datapreview.extension.util.PreviewRecordTransformer	transformation	undefined
]

And here are the plugins that it does load:

$ kubectl logs -n dev confluent-connect-0 | grep -E 'Scanning for plugin classes| Added plugin' | sed -r 's/^(.+INFO Added plugin.+):PluginDesc.+/\1/'
Defaulted container "confluent-connect" out of: confluent-connect, config-init-container (init)
[2025-01-27 17:15:32,510] INFO Scanning for plugin classes. This might take a moment ... (org.apache.kafka.connect.cli.AbstractConnectCli:127)
[2025-01-27 17:17:03,822] INFO Added plugin 'org.apache.kafka.connect.converters.FloatConverter'
[2025-01-27 17:17:03,822] INFO Added plugin 'org.apache.kafka.connect.transforms.ReplaceField$Key'
[2025-01-27 17:17:03,822] INFO Added plugin 'org.apache.kafka.connect.transforms.Filter'
[2025-01-27 17:17:03,822] INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorSourceConnector'
[2025-01-27 17:17:03,822] INFO Added plugin 'org.apache.kafka.connect.transforms.InsertField$Value'
[2025-01-27 17:17:03,822] INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampConverter$Key'
[2025-01-27 17:17:03,822] INFO Added plugin 'org.apache.kafka.connect.converters.DoubleConverter'
[2025-01-27 17:17:03,823] INFO Added plugin 'org.apache.kafka.connect.connector.policy.AllConnectorClientConfigOverridePolicy'
[2025-01-27 17:17:03,823] INFO Added plugin 'io.confluent.kafka.secretregistry.client.config.provider.SecretConfigProvider'
[2025-01-27 17:17:03,823] INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampRouter'
[2025-01-27 17:17:03,823] INFO Added plugin 'org.apache.kafka.connect.transforms.RegexRouter'
[2025-01-27 17:17:03,823] INFO Added plugin 'org.apache.kafka.connect.transforms.HoistField$Value'
[2025-01-27 17:17:03,823] INFO Added plugin 'org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicy'
[2025-01-27 17:17:03,823] INFO Added plugin 'org.apache.kafka.connect.transforms.ValueToKey'
[2025-01-27 17:17:03,823] INFO Added plugin 'org.apache.kafka.connect.transforms.DropHeaders'
[2025-01-27 17:17:03,823] INFO Added plugin 'org.apache.kafka.connect.transforms.Cast$Key'
[2025-01-27 17:17:03,823] INFO Added plugin 'org.apache.kafka.connect.storage.SimpleHeaderConverter'
[2025-01-27 17:17:03,823] INFO Added plugin 'org.apache.kafka.connect.converters.LongConverter'
[2025-01-27 17:17:03,823] INFO Added plugin 'org.apache.kafka.common.config.provider.FileConfigProvider'
[2025-01-27 17:17:03,823] INFO Added plugin 'org.apache.kafka.connect.transforms.InsertHeader'
[2025-01-27 17:17:03,824] INFO Added plugin 'io.confluent.connect.json.JsonSchemaConverter'
[2025-01-27 17:17:03,824] INFO Added plugin 'org.apache.kafka.common.config.provider.DirectoryConfigProvider'
[2025-01-27 17:17:03,824] INFO Added plugin 'org.apache.kafka.connect.transforms.Flatten$Key'
[2025-01-27 17:17:03,824] INFO Added plugin 'io.confluent.connect.rest.datapreview.extension.util.PreviewRecordTransformer'
[2025-01-27 17:17:03,824] INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorCheckpointConnector'
[2025-01-27 17:17:03,824] INFO Added plugin 'org.apache.kafka.connect.transforms.HeaderFrom$Value'
[2025-01-27 17:17:03,824] INFO Added plugin 'org.apache.kafka.connect.transforms.SetSchemaMetadata$Key'
[2025-01-27 17:17:03,824] INFO Added plugin 'io.confluent.kafka.schemaregistry.client.config.provider.SchemaRegistryConfigProvider'
[2025-01-27 17:17:03,824] INFO Added plugin 'org.apache.kafka.connect.json.JsonConverter'
[2025-01-27 17:17:03,824] INFO Added plugin 'org.apache.kafka.connect.transforms.predicates.TopicNameMatches'
[2025-01-27 17:17:03,824] INFO Added plugin 'org.apache.kafka.connect.transforms.ReplaceField$Value'
[2025-01-27 17:17:03,824] INFO Added plugin 'org.apache.kafka.connect.transforms.SetSchemaMetadata$Value'
[2025-01-27 17:17:03,824] INFO Added plugin 'org.apache.kafka.connect.converters.IntegerConverter'
[2025-01-27 17:17:03,825] INFO Added plugin 'org.apache.kafka.connect.transforms.HeaderFrom$Key'
[2025-01-27 17:17:03,825] INFO Added plugin 'io.confluent.connect.protobuf.ProtobufConverter'
[2025-01-27 17:17:03,825] INFO Added plugin 'org.apache.kafka.connect.transforms.predicates.HasHeaderKey'
[2025-01-27 17:17:03,826] INFO Added plugin 'org.apache.kafka.connect.transforms.predicates.RecordIsTombstone'
[2025-01-27 17:17:03,826] INFO Added plugin 'org.apache.kafka.connect.transforms.MaskField$Value'
[2025-01-27 17:17:03,826] INFO Added plugin 'org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy'
[2025-01-27 17:17:03,826] INFO Added plugin 'org.apache.kafka.common.config.provider.EnvVarConfigProvider'
[2025-01-27 17:17:03,826] INFO Added plugin 'io.confluent.connect.rest.datapreview.extension.ConnectorDataPreviewRestExtension'
[2025-01-27 17:17:03,826] INFO Added plugin 'org.apache.kafka.connect.storage.StringConverter'
[2025-01-27 17:17:03,826] INFO Added plugin 'org.apache.kafka.connect.transforms.MaskField$Key'
[2025-01-27 17:17:03,826] INFO Added plugin 'org.apache.kafka.connect.converters.ByteArrayConverter'
[2025-01-27 17:17:03,826] INFO Added plugin 'io.confluent.connect.avro.AvroConverter'
[2025-01-27 17:17:03,826] INFO Added plugin 'io.confluent.connect.security.ConnectSecurityExtension'
[2025-01-27 17:17:03,826] INFO Added plugin 'org.apache.kafka.connect.transforms.Cast$Value'
[2025-01-27 17:17:03,826] INFO Added plugin 'org.apache.kafka.connect.transforms.ExtractField$Key'
[2025-01-27 17:17:03,827] INFO Added plugin 'org.apache.kafka.connect.transforms.Flatten$Value'
[2025-01-27 17:17:03,827] INFO Added plugin 'org.apache.kafka.connect.transforms.InsertField$Key'
[2025-01-27 17:17:03,827] INFO Added plugin 'org.apache.kafka.connect.converters.BooleanConverter'
[2025-01-27 17:17:03,827] INFO Added plugin 'org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension'
[2025-01-27 17:17:03,827] INFO Added plugin 'org.apache.kafka.connect.transforms.ExtractField$Value'
[2025-01-27 17:17:03,827] INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector'
[2025-01-27 17:17:03,827] INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampConverter$Value'
[2025-01-27 17:17:03,827] INFO Added plugin 'org.apache.kafka.connect.transforms.HoistField$Key'
[2025-01-27 17:17:03,827] INFO Added plugin 'org.apache.kafka.connect.converters.ShortConverter'

dtroiano · 27 January 2025 17:46

Do any differences jump out if you compare to this demo? This is what I used to test.

binnsr · 27 January 2025 17:59

The only thing that jumps out at me is image versions…
that demo uses

  image:
    application: confluentinc/cp-server-connect:7.7.0
    init: confluentinc/confluent-init-container:2.9.2

and I’m using:

  image:
    application: confluentinc/cp-server-connect:latest
    init: confluentinc/confluent-init-container:2.10.0

where latest is 7.8.0

We’re also connected to an external Kafka via mtls, and we’re not specifying a SchemaRegistry or the RESTProxy, but I don’t think that’s an issue. We do have the RESTProxy deployed separately - using the same helm chart, but different deployment.
I can provide my full chart if you want to review that

binnsr · 27 January 2025 18:05

Also, I do not appear to have DM permissions in this system.

dtroiano · 27 January 2025 20:07

Oops, I hadn’t realized that we intentionally turned this off due to spam issues. I tried DMing you since you might be able to reply to me, or the other option is to join the Confluent Community Slack and DM me there.

dtroiano · 27 January 2025 20:08

I don’t think this is the issue – I upgraded the demo to cp-server-connect:latest / confluent-init-container:2.10.0 and it still works as expected.

Yes please share it

binnsr · 27 January 2025 20:55

Here is a public copy for your reference.

dtroiano · 27 January 2025 21:29

This line is suspicious since the plugin is installed at /mnt/plugins:

plugin.path=/usr/share/java,/usr/share/confluent-hub-components

I’d also double check that the plugin is there, e.g., kubectl exec -it --namespace confluent connect-0 -- bash and make sure that the Datagen connector is present under /mnt/plugins/.

binnsr · 27 January 2025 21:39

Well that was a dumb leftover from earlier attempts to get it running.
That resolves this particular issue - and I think the only remaining one here is because my datagen topic isn’t configured.

thank you for your assistance!

system · 26 February 2025 21:39

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Controlcenter, Connect pods are failing Containers	6	35	10 September 2024
Kafka Connect could not laod config provider class `InternalSecretConfigProvider ` Kafka Connect	3	2156	19 August 2023
Kafka connect s3 sink not able to create connectors Kafka Connect	3	3575	27 January 2022
Error Deploying ServiceNow Sink Kafka Connect	3	3034	25 February 2022
Stand alone cp kafka connect to confluent cloud Kafka Connect	2	2667	28 February 2023

Confluent for Kubernetes - unable to setup plugins

Related topics