How can I externalize the connector configurations?

Kafka Connect
1

I am trying to store all the environment specific configuration values of Kafka connect in a centralized ConfigProvider that I can refer to

When I tried to use external configurations while creating the connectors, it is resolving the values literally instead of interpreting the values just like the connect-distributed binary would .

Is there any way to accomplish this? Can the secrets be stored perhaps as environment variables?

Thanks!

2

Kafka Connect 3.5.0 finally added an environment variable ConfigProvider.

What version are you using? What specific issues are you having? Can you share your configs?

3

More than an issue it is a question.

Say you have the following configuration for a connector :

{
  "config": {
    "name": "test-couchbase-source",
    "connector.class": "com.couchbase.connect.kafka.CouchbaseSourceConnector",
    "tasks.max": "1",
    "couchbase.topic": "test-default",
    "couchbase.seed.nodes": "couchbase",
    "couchbase.bootstrap.timeout": "10s",
    "couchbase.bucket": "travel-sample",
    "couchbase.username": "couchbase",
    "couchbase.password": "couchbase",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "couchbase.source.handler": "com.couchbase.connect.kafka.handler.source.RawJsonSourceHandler",
    "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
    "couchbase.event.filter": "com.couchbase.connect.kafka.filter.AllPassFilter",
    "couchbase.stream.from": "BEGINNING",
    "couchbase.compression": "ENABLED",
    "couchbase.flow.control.buffer": "16m",
    "couchbase.persistence.polling.interval": "100ms"
  }
}

How can I externalize the configuration for the following secrets?

    "couchbase.username": "couchbase",
    "couchbase.password": "couchbase",

I tried to use the same syntax as the other configuration options using a custom ConfigProvider, but that did not seem to work as the values were resolved literally. For example :

config.providers=file
config.providers.file.class=org.apache.kafka.common.config.provider.FileConfigProvider

And

    "couchbase.username:${file:/opt/connect-secrets.properties:couchbase.username}

Was resolved literally to the value :${file:/opt/connect-secrets.properties:couchbase.username}

Can ConfigProvider be used to resolve the connectors configuration? Or, what can be used?

4

That seems correct to me. Make sure you’re running connect version over 2.0 where that feature was added and that the Connect process user has read access to the file

https://docs.confluent.io/platform/current/connect/security.html#externalize-secrets

5

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.