External connection is not working using SASL_SSL protocol on Confluent operator

Ops Containers
1

I am using confluent operator for Kubernetes cluster. I am using the following manifest file to run kafka, zookeeper, schema-registry.

apiVersion: platform.confluent.io/v1beta1
kind: Zookeeper
metadata:
  name: zookeeper
spec:
  replicas: 3
  image:
    application: confluentinc/cp-zookeeper:7.3.0
    init: confluentinc/confluent-init-container:2.5.0
  dataVolumeCapacity: 10Gi
  logVolumeCapacity: 10Gi
---
apiVersion: platform.confluent.io/v1beta1
kind: Kafka
metadata:
  name: kafka
spec:
  replicas: 3
  image:
    application: confluentinc/cp-server:7.3.0
    init: confluentinc/confluent-init-container:2.5.0
  dataVolumeCapacity: 10Gi
  tls:
    secretRef: tls-kafka
  listeners:
    external:
      authentication:
        type: plain
        jaasConfigPassThrough:
          secretRef: jaas-credential
      tls:
        enabled: false
      externalAccess:
        type: loadBalancer
        loadBalancer:
          domain: 12.214.3.123
          advertisedPort: 29093
  dependencies:
    zookeeper:
      endpoint: zookeeper:2181
---
apiVersion: platform.confluent.io/v1beta1
kind: SchemaRegistry
metadata:
  name: schemaregistry
spec:
  replicas: 1
  image:
    application: confluentinc/cp-schema-registry:7.3.0
    init: confluentinc/confluent-init-container:2.5.0
  externalAccess:
    type: loadBalancer
    loadBalancer:
      domain: 12.214.3.123
      prefix: sr
  dependencies:
    kafka:
      bootstrapEndpoint: kafka:9071

As I have not added any internal authentication, I am able to run the below commands inside the container.

kafka-topics --create  --topic kafka_test --bootstrap-server kafka.confluent.svc.cluster.local:9071

kafka-console-producer --broker-list kafka.confluent.svc.cluster.local:9071 --topic kafka_test

kafka-console-consumer --broker-list kafka.confluent.svc.cluster.local:9071 --topic kafka_test --from-beginning

jaas config

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
   username="admin" \
   password="admin-secret";

But I am not able to connect from outside. Basically, external authentication is not working

kafka-console-consumer --bootstrap-server kafka.confluent.svc.cluster.local:29093 --topic kafka_test --consumer-property security.protocol=SASL_PLAINTEXT --consumer-property sasl.mechanism=PLAIN --consumer.config /opt/confluentinc/plain-jaas.conf --from-beginning

I tried to run the above command inside the container with jaas config. It is throwing the below error.

[2023-01-10 13:31:28,341] WARN [Consumer clientId=console-consumer, groupId=console-consumer-44133] Connection to node -1 (kafka.confluent.svc.cluster.local/10.42.2.144:29093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
2

hey @manoj-quartic welcome :slight_smile:

is kafka.confluent.svc.cluster.local resolvable from outside the K8s cluster?
looks like an internal hostname/DNS

did you try the same command with port 9071?

Best,
Michael

3

I am trying from inside the container only to simulate the outside access. I used the load balancer IP (external IP) as well.

From the outside, I am getting some errors as well

kafka-console-consumer --bootstrap-server 12.214.3.123:29093 --topic kafka_test --consumer-property security.protocol=SASL_PLAINTEXT --consumer-property sasl.mechanism=PLAIN --consumer.config /opt/confluentinc/plain-jaas.conf --from-beginning

%3|1673423270.443|FAIL|rdkafka#producer-1| [thrd:sasl_ssl://b1.12.214.3.123:29093/bootstrap]: sasl_ssl://b1.12.214.3.123:29093/bootstrap: Failed to resolve 'b1.12.214.3.123:9092': nodename nor servname provided, or not known (after 51ms in state CONNECT)

I think it is trying to resolve the prefix which is added by the external load balancer configuration in Kafka. I am a bit confused about accessing the 3 replicas of kafka from outside of the cluster.

4

ok I see
did you check the following:

https://docs.confluent.io/operator/current/co-loadbalancers.html#configure-load-balancers-to-access-confluent-components

best,
michael