How to Safely Run Multiple Kafka Connect Pods on Kubernetes Using Strimzi?

Kafka Connect Managed Connectors
1

Hello community, how are you?

I’m new to Kafka and still learning. But I’m working on deploying Kafka Connect.

A question came up.

I’m deploying my Kafka Connect on Kubernetes and want it to have at least two pods running.

I saw some required fields in this documentation:

I have the following questions:

The replicas must have the same group.id to be part of the same cluster, right?

In this documentation,

"Unless you modify these default settings, each instance connecting to the same Kafka cluster is deployed with the same values. In practice, this means all instances form a cluster and use the same internal topics.

Multiple instances attempting to use the same internal topics will cause unexpected errors, so you must change the values ​​of these properties for each instance.
Multiple instances attempting to use the same internal topics will cause unexpected errors, so you must change the values ​​of these properties for each instance."

Does this mean that if I run two pods with the same group.id and the same topic configuration, there could be a problem?

I’m quite confused. Could someone explain to me how to run two or more Kafka Connect pods on Kubernetes without problems?

Thank you very much for your attention.

2

hey @jpfsemp

welcome :slight_smile:

I’m not fully in the details of strimzi, though having 2 pods/workers should not harm your setup.

kafka connect is designed to run in a distributed way on several nodes/servers.

therefore it should be possible to just increase the replicas to a value greater than “1!

check also this one here:
https://itnext.io/step-by-step-guide-deploying-kafka-connect-via-strimzi-operator-on-kubernetes-6357c123abe9

hth,

michael

3

Thank you so much for your feedback.
It helped me a lot. Good luck!!