Kafka connect container stops

Kafka Connect
1

Hi ,

I have installed kafka connect container in distributed mode. provided the bootstrap server url ( kafka broker is running on confluent cloud). Kafka connect runs for a while and shuts down. Following is excerpts from logs that seem to point out the problem but am unsure of the solution or missing configurations.

[2022-02-08 09:43:39,478] INFO Unable to use admin client to verify the cleanup policy of 'test-cp-kafka-connect-status' topic is 'compact', either because the broker is an older version or because the Kafka principal used for Connect internal topics does not have the required permission to describe topic configurations. (org.apache.kafka.connect.util.TopicAdmin)

(org.apache.kafka.clients.producer.ProducerConfig)

[2022-02-08 09:43:39,482] WARN The configuration 'group.id' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2022-02-08 09:43:39,482] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2022-02-08 09:43:39,482] WARN The configuration 'internal.key.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2022-02-08 09:43:39,482] WARN The configuration 'metrics.context.connect.kafka.cluster.id' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2022-02-08 09:43:39,482] WARN The configuration 'status.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2022-02-08 09:43:39,482] WARN The configuration 'offset.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2022-02-08 09:43:39,482] WARN The configuration 'value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2022-02-08 09:43:39,482] WARN The configuration 'key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2022-02-08 09:43:39,482] WARN The configuration 'config.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2022-02-08 09:43:39,482] WARN The configuration 'metrics.context.connect.group.id' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2022-02-08 09:43:39,482] WARN The configuration 'rest.advertised.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2022-02-08 09:43:39,482] WARN The configuration 'status.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2022-02-08 09:43:39,482] WARN The configuration 'rest.port' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2022-02-08 09:43:39,482] WARN The configuration 'config.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2022-02-08 09:43:39,482] WARN The configuration 'key.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2022-02-08 09:43:39,482] WARN The configuration 'internal.key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2022-02-08 09:43:39,482] WARN The configuration 'value.converter.schema.registry.url' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2022-02-08 09:43:39,482] WARN The configuration 'internal.value.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2022-02-08 09:43:39,482] WARN The configuration 'value.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2022-02-08 09:43:39,482] WARN The configuration 'internal.value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2022-02-08 09:43:39,482] WARN The configuration 'offset.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2022-02-08 09:43:39,482] WARN The configuration 'key.converter.schema.registry.url' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)

[2022-02-08 09:43:39,539] ERROR [Worker clientId=connect-1, groupId=test] Uncaught exception in herder work thread, exiting:  (org.apache.kafka.connect.runtime.distributed.DistributedHerder)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [test-cp-kafka-connect-status]

Appreciate for your insights.

2

Hi @Shahed

seems to related to an authorization issue

I guess there is no acl which allows topic access/topic creation

have a look at

https://docs.confluent.io/platform/current/tutorials/cp-demo/docs/hybrid-cloud.html#

and

https://docs.confluent.io/platform/current/tutorials/examples/ccloud-observability/docs/producer-scenarios/authorization-problem.html

hth,
michael

3

@Shahed within the Cloud web UI / Console, if you go to the menu and choose Data Integrations->Connectors, then select your connector it can walk you through configuring security configurations for your connector. It can be tricky to translate the generated configuration to the docker environment variables, but this documentation may help with that step: Docker Configuration Parameters for Confluent Platform | Confluent Documentation.

Generally you need to prefix the each configuration as the environment variable with CONNECT_ and then replace each . in the generated config with an underscore (_).

Hope this helps

1 Like
4

@Shahed here are a few more resources which may help as well:

1 Like
5

Thanks Rick and Michael for response. I am checking ACL’s and if principal used by connect to cluster has sufficient privileges.

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.