Self managed Connectors on K8s

Kafka Connect Self-Managed Connectors
1

If you’re looking for an example of running a distributed Connect cluster in Kubernetes, check out this example which is part of a broader DevOps project over at Confluent.

https://docs.confluent.io/platform/current/tutorials/kafka-devops/case-studies/connector-management.html

The example includes a custom Docker image which installs the necessary connectors and is published to Docker Hub:

FROM confluentinc/cp-server-connect:5.5.1

RUN confluent-hub install --no-prompt confluentinc/kafka-connect-jdbc:5.5.1
RUN confluent-hub install --no-prompt confluentinc/kafka-connect-elasticsearch:5.5.1

RUN wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.21/mysql-connector-java-8.0.21.jar -P /usr/share/java/kafka-connect-jdbc/
...

Connect is fronted by a k8s Service which provides a network abstraction in front of the Connect REST service for load balancing and location transparency:

apiVersion: v1
kind: Service
metadata:
  name: connect
  labels:
    app: connect-service
spec:
  selector:
    app: connect-service
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8083

Which means we can interact with the service using a very friendly URL:

bash-5.0# curl -s http://connect/connectors | jq
[
  "jdbc-customers"
]

The Connect workers are deployed in a k8s Deployment providing a scalable and fault tolerant cluster of workers. See the project source for the full details…

apiVersion: apps/v1
kind: Deployment
metadata:
  name: connect-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: connect-service
  template:
    metadata:
      labels:
        app: connect-service
...

The entire project utilizes a GitOps approach to management of resources… Check out this blog for the full details…

2 Likes