I am trying to externalize connector’s properies using the FileConfigProvider. I am extending the connect image confluentinc/cp-kafka-connect:7.5.0.
I keep getting the error “error_code”:500,“message”:"Unexpected character (‘$’ (code 36)) when the instance of the connectors is created.
It seems connect cant understand or find the variables defined in the connect-secrets.properties file.
Here are the relevant docker file commands:
FROM confluentinc/cp-kafka-connect:7.5.0
External secrets config
ENV CONNECT_CONFIG_PROVIDERS: ‘file’
ENV CONNECT_CONFIG_PROVIDERS_FILE_CLASS: ‘org.apache.kafka.common.config.provider.FileConfigProvider’
#copy protected variables into an image folder
COPY ./prot/connect-secrets.properties /temp/connectors/connect-secrets.properties
#copy container start-up script
COPY ./commadscript.sh /usr/local/bin/commadscript.sh
USER appuser
#run script to launch kafka-connect and create connector instances
CMD /usr/local/bin/commadscript.sh
In the commandscript.sh file i instantiate the connectors
curl -s -X PUT -H “Content-Type:application/json” http://localhost:8083/connectors/splunk-sink-confluent-audit/config
-d '{
“name”: “splunk-sink-confluent-audit”,
“connector.class”: “com.splunk.kafka.connect.SplunkSinkConnector”,
“topics”: “confluent-audit-log-events”,
“splunk.hec.token”:“${file:/temp/connectors/connect-secrets.properties:SPLUNK_TOKEN}”,
“splunk.hec.uri”:“${file:/temp/connectors/connect-secrets.properties:SPLUNK_URI}”,
“splunk.indexes”:“${file:/temp/connectors/connect-secrets.properties:SPLUNK_INDEXES}”,
…}
curl -s -X PUT -H “Content-Type:application/json” http://localhost:8083/connectors/lambda-costcenter-consumer/config
-d '{
“name”: “lambda-costcenter-consumer”,
“connector.class”: “io.confluent.connect.aws.lambda.AwsLambdaSinkConnector”,
“tasks.max”: 1,
“topics”: “${file:/temp/connectors/connect-secrets.properties:CONSUMED_TOPICS}”,
“input.data.format”: “AVRO”,
“behavior.on.error”: “fail”,
“aws.credentials.provider.class”: “com.amazonaws.auth.InstanceProfileCredentialsProvider”,
“aws.lambda.function.name”: “${file:/temp/connectors/connect-secrets.properties:LAMBDA_ARN}”,
…
}
Here is what i see in the container logs
[2024-01-18 21:56:09,085] INFO SplunkSinkConnectorConfig values:
enable.timestamp.extraction = false
kerberos.keytab.path =
kerberos.user.principal =
splunk.flush.window = 30
splunk.header.custom =
splunk.header.host = splunk.header.host
splunk.header.index = splunk.header.index
splunk.header.source = splunk.header.source
splunk.header.sourcetype = ccloud_audit_logs
splunk.header.support = false
splunk.hec.ack.enabled = false
splunk.hec.ack.poll.interval = 10
splunk.hec.ack.poll.threads = 2
splunk.hec.backoff.threshhold.seconds = 60
splunk.hec.enable.compression = false
splunk.hec.event.timeout = 300
splunk.hec.http.keepalive = true
splunk.hec.json.event.enrichment =
splunk.hec.json.event.formatted = false
splunk.hec.lb.poll.interval = 120
splunk.hec.max.batch.size = 500
splunk.hec.max.http.connection.per.channel = 2
splunk.hec.max.outstanding.events = 1000000
splunk.hec.max.retries = -1
splunk.hec.raw = false
splunk.hec.raw.line.breaker =
splunk.hec.socket.timeout = 60
splunk.hec.ssl.trust.store.password = [hidden]
splunk.hec.ssl.trust.store.path =
splunk.hec.ssl.trust.store.type = JKS
splunk.hec.ssl.validate.certs = false
splunk.hec.threads = 1
splunk.hec.token = [hidden]
splunk.hec.total.channels = 2
splunk.hec.track.data = true
splunk.hec.uri = ${file:/temp/connectors/connect-secrets.properties:SPLUNK_URI}
splunk.hec.use.record.timestamp = true
splunk.indexes = ${file:/temp/connectors/connect-secrets.properties:SPLUNK_INDEXES}
splunk.sources =
splunk.validation.disable = false
2024-01-18 16:56:09 (com.splunk.kafka.connect.SplunkSinkConnectorConfig:369)
2024-01-18 16:56:09 [2024-01-18 21:56:09,104] ERROR Uncaught exception in REST call to /connectors/splunk-sink-confluent-audit/config (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper:64)
2024-01-18 16:56:09 java.lang.IllegalArgumentException: Illegal character in scheme name at index 0: ${file:/temp/connectors/connect-secrets.properties:SPLUNK_URI}/services/collector
2024-01-18 16:56:09 at java.base/java.net.URI.create(URI.java:883)
…
Caused by: java.net.URISyntaxException: Illegal character in scheme name at index 0: ${file:/temp/connectors/connect-secrets.properties:SPLUNK_URI}/services/collector
2024-01-18 16:56:09 at java.base/java.net.URI$Parser.fail(URI.java:2913)