Kafaka Connect same clusterId for different group.id

Kafka Connect
1

Hi there,

I am a bit confused about getting the same connect clusterIds with different group.id’s.
I started 3x containers, with the same Dockerfile:

FROM confluentinc/cp-kafka-connect:latest
RUN confluent-hub install --no-prompt mongodb/kafka-connect-mongodb:1.7.0
ENV CONNECT_PLUGIN_PATH="/usr/share/java,/usr/share/confluent-hub-components"

Then in docker compose file, two of them I specified CONNECT_GROUP_ID: cluster1 and the other cluster2, like following:

connect1:
    image: cdc-tutorial-connect-1.7.0:1.0
    build:
      context: .
      dockerfile: Dockerfile-MongoConnect
    hostname: connect1
    container_name: connect1
    depends_on:
      - zookeeper
      - broker
    networks:
      - localnet
    environment:
      CONNECT_BOOTSTRAP_SERVERS: "broker:29092"
      CONNECT_REST_ADVERTISED_HOST_NAME: connect1
      CONNECT_REST_PORT: 8083
      CONNECT_GROUP_ID: cluster1
      CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000
      CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_ZOOKEEPER_CONNECT: "zookeeper:2181"
      CONNECT_PLUGIN_PATH: "/usr/share/java,/usr/share/confluent-hub-components"
      CONNECT_AUTO_CREATE_TOPICS_ENABLE: "true"
      CONNECT_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"

I verified this is taking effect:

C:\>docker exec -it connect1 /bin/env |findstr CONNECT_GROUP_ID
CONNECT_GROUP_ID=cluster1

C:\>docker exec -it connect2 /bin/env |findstr CONNECT_GROUP_ID
CONNECT_GROUP_ID=cluster1

C:\>docker exec -it connect3 /bin/env |findstr CONNECT_GROUP_ID
CONNECT_GROUP_ID=cluster2

C:\>docker exec -it connect1 /bin/grep group.id /etc/kafka-connect/kafka-connect.properties
group.id=cluster1

C:\>docker exec -it connect2 /bin/grep group.id /etc/kafka-connect/kafka-connect.properties
group.id=cluster1

C:\>docker exec -it connect3 /bin/grep group.id /etc/kafka-connect/kafka-connect.properties
group.id=cluster2

I would expect there will be 2 different cluster IDs (1 and 2 belongs to one, 3 the other), but they have the same cluster ID:

[MongoDB Kafka Connector Tutorial] : for i in 1 2 3
> do
> curl -s http://connect${i}:8083 -w '\n'
> done
{"version":"7.1.0-ccs","commit":"c86722379ab997cc","kafka_cluster_id":"cBDDW609RIqVBogWFoKqeg"}
{"version":"7.1.0-ccs","commit":"c86722379ab997cc","kafka_cluster_id":"cBDDW609RIqVBogWFoKqeg"}
{"version":"7.1.0-ccs","commit":"c86722379ab997cc","kafka_cluster_id":"cBDDW609RIqVBogWFoKqeg"}

Did I misunderstand the relationship between cluster Id and group.id? Should’nt connect3 have a different cluster ID?

Thanks,
Sean

2

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.