I want to duplicate Connect Server, but an error occurs

Kafka Connect
1

I want to configure two connect servers. I wrote the compose below, but it works normally when registering the connector through connect-1, but an error occurs when registering the connector through connect-2.

{“error_code”:500,“message”:“IO Error trying to forward REST request: java.net.ConnectException: Connection refused”}%

Connectors registered through connect-1 can also be viewed in connect-2, and from what I have checked, I know that the source connector must be registered separately for connect-1 and connect-2.
I want to hear an answer

The host information registered in extra_hosts is registered in CONNECT_REST_ADVERTISED_HOST_NAME, but it was not registered due to the URL when writing the post, so it was modified.

I need your help

docker compose

version: ‘3.0’

services:
connect-1:
build: .
image: comus-connect
container_name: connect-1
ports:
- “8083:8083”
networks:
- kn
restart: always
environment:
- CONNECT_PLUGIN_PATH=“/usr/share/java”
- CONNECT_BOOTSTRAP_SERVERS=kafka.plgrim.com:8120
- CONNECT_GROUP_ID=connect-group
- CONNECT_CONFIG_STORAGE_TOPIC=KAFKA_CONNECT_CONFIGS
- CONNECT_OFFSET_STORAGE_TOPIC=KAFKA_CONNECT_OFFSETS
- CONNECT_STATUS_STORAGE_TOPIC=KAFKA_CONNECT_STATUS
- CONNECT_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
- CONNECT_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
- CONNECT_INTERNAL_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
- CONNECT_INTERNAL_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter

  • CONNECT_REST_ADVERTISED_HOST_NAME=connect-1

    • CONNECT_REST_ADVERTISED_PORT=8083
      • CONNECT_REST_PORT=8083
      • CONNECT_LISTENERS=http://:8083
        extra_hosts:
      • “kafka.plgrim.com:172.19.0.2”
      • “connect-1.plgrim.com:127.0.0.1”
      • “connect-2.plgrim.com:172.19.0.7”

    connect-2:
    build: .
    image: comus-connect
    container_name: connect-2
    ports:
    - “8084:8084”
    networks:
    - kn
    restart: always
    environment:
    - CONNECT_PLUGIN_PATH=“/usr/share/java”
    - CONNECT_BOOTSTRAP_SERVERS=kafka.plgrim.com:8120
    - CONNECT_GROUP_ID=connect-group
    - CONNECT_CONFIG_STORAGE_TOPIC=KAFKA_CONNECT_CONFIGS
    - CONNECT_OFFSET_STORAGE_TOPIC=KAFKA_CONNECT_OFFSETS
    - CONNECT_STATUS_STORAGE_TOPIC=KAFKA_CONNECT_STATUS
    - CONNECT_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
    - CONNECT_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
    - CONNECT_INTERNAL_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
    - CONNECT_INTERNAL_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
    CONNECT_REST_ADVERTISED_HOST_NAME=connect-2

    • CONNECT_REST_ADVERTISED_PORT=8084
      - CONNECT_REST_PORT=8084
      - CONNECT_LISTENERS=http://:8084
      extra_hosts:
      - “kafka.plgrim.com:172.19.0.2”
      - “connect-1.plgrim.com:127.0.0.1”
      - “connect-2.plgrim.com:172.19.0.7”

networks:
kn:
name: kn
driver: bridge
external: true

2

the source connector must be registered separately for connect-1 and connect-2.

This is incorrect. The request will be forwarded to the appropriate leader worker, then tasks will be distributed as necessary.

Perhaps you’d have better luck scaling 2+ replicas using Strimzi or Confluent CFK in minikube? That way, you won’t need to duplicate config or worry about advertised listeners between workers

Otherwise, extra hosts isn’t what you should be using between two compose services, nor ip addresses

I have an example compose file at GitHub - OneCricketeer/apache-kafka-connect-docker: [KAFKA-9774] Un-official Docker Image for Apache Kafka Connect

3

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.