Hi all,
i’m asking how to create connector on kafka connect in distributed mode.
Right now we have kafka connect deploy on k8s with 3 replicas; we have an init script for each replica that create connectors.
Is it right to create connectors in each replica or is better to user k8s service that stay in front of replicas and send api request randomly to replicas?
Which way the best approach to create connector on kafka connect deployed in k8s?
Thanks
Hi @rtrive , Did you have a look at Confluent Operator? I think you might find it useful to make your life easier in keeping Confluent running on k8s.
Having a look at the architecture in the docs page, the approach would be to have your pods sit behind Load Balancer, and you would just use one address from your code
In terms of the automation, I guess one option would be to separate the deployment of connectors from the actual connect Images. In the connect pods you will need to install the connectors, sure, or use an image with the connectors already installed. But you could leave it to another pod (a configuration one) the task to create the connectors using the REST API of Connect.
Have a look at this great post from @rick : Self managed Connectors on K8s
Let us know if you need more details!
Bear in mind that Kafka Connect in distributed mode uses Kafka itself to store config and status of connectors. So once you’ve provisioned the connect cluster, you create your connectors once and then as workers join and leave the cluster they pick up their config from the internal topics stored on Kafka itself.
@gianlucanatali, yes we saw confluent operator but we’re not using right now because it’s not free and i need to stay with free software. But thanks i will consider to deploy a separate container only for configuration 
@rmoff ok now it’s more clear, it useless create connectors for each pod, it’s better to create once for all. Thanks
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.
