Unable to Connect to Kafka Brokers deployed on Kubernetes

Ops Containers
1

0

I am trying to Deploy Kafka Cluster with 3 Brokers & 1 Zookeeper Service & pod on Kubernetes cluster(EKS) . I also deployed Minio on the same Cluster. Please find my Kafka & Zoo-keeper files

kind: Deployment
apiVersion: apps/v1
metadata:
  name: zookeeper-service
spec:
  selector:
    matchLabels:
      app: zookeeper
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      containers:
      - name: zoo1
        image: docker.io/zookeeper
        ports:
        - containerPort: 2181
        env:
        - name: ZOOKEEPER_ID
          value: "1"
        - name: ZOOKEEPER_SERVER_1
          value: zoo1
---
apiVersion: v1
kind: Service
metadata:
  name: zoo1
  labels:
    app: zookeeper-1
spec:
  ports:
  - name: client
    port: 2181
    protocol: TCP
  - name: follower
    port: 2888
    protocol: TCP
  - name: leader
    port: 3888
    protocol: TCP
  selector:
    app: zookeeper-1

---
apiVersion: v1
kind: Service
metadata:
  name: kafka-service 
spec:
  type: NodePort 
  selector:
    component: kafka
  ports:
  - name: kafka
    port: 9092
    targetPort: 9092
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
  
spec:
  selector:
    matchLabels:
      app: kafka
  serviceName: "kafka"
  replicas: 3
  podManagementPolicy: OrderedReady
  template:
    metadata:
      labels:
        app: kafka # has to match .spec.selector.matchLabels
    spec:
      containers:
      - name: kafka
        image: wurstmeister/kafka:2.11-2.0.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9092
          name: plaintext
        - containerPort: 9999
          name: jmx
        env:
          - name: KAFKA_ADVERTISED_PORT
            value: "9092"
          - name: BROKER_ID_COMMAND
            value: "hostname | cut -d'-' -f2"
          - name: KAFKA_ZOOKEEPER_CONNECT
            value: "zookeeper-service:2181"
          - name: KAFKA_LISTENERS
            value: "PLAINTEXT://:9092"
          - name: KAFKA_JMX_OPTS
            value: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.rmi.port=9999 -Djava.rmi.server.hostname=127.0.0.1"
          - name: JMX_PORT
            value: "9999"

I am trying to Publish Notifications from Minio to Kafka using this reference Minio-Kafka

But always i am getting this error “mc.exe: Unable to set ‘notify_kafka:vpd-kafka brokers=172.17.0.20:9092 topic=test’ to server: kafka: client has run out of available brokers to talk to (Is your cluster reachable?).”

I Researched on the same error & tried differnet ways of Deploying kafka brokers as Load balancers/Nodeport’s But nothing seems work. Kindly Suggest what am i doing wrong.

Thank you in Advance

2

Hi,

all pods are running fine?

what’s the output of kubectl get svc
did you check advertised listener settings?