What is wrong with my cluster setup?

programista4k · 15 December 2021 22:37

Hello frens,
I want to run kafka cluster with docker compose. It should consist of 3 kafka brokers.
This is how it looks like

version: '3.8'
services:

  zookeeper-1:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper-1
    ports:
      - '12181:2181'
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  zookeeper-2:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper-2
    ports:
      - '22181:2181'
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      
  zookeeper-3:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper-3
    ports:
      - '32181:2181'    
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      
  kafka-1:
    image: confluentinc/cp-kafka:latest
    container_name: kafka-1
    ports:
      - '19092:19092'
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-1:9092,PLAINTEXT_HOST://localhost:19092
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      BOOTSTRAP_SERVERS: kafka1:19092,kafka2:29092,kafka3:39092
    depends_on:
      - zookeeper-1
      - zookeeper-2
      - zookeeper-3
      
  kafka-2:
    image: confluentinc/cp-kafka:latest
    container_name: kafka-2
    ports:
      - '29092:29092'
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-2:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      BOOTSTRAP_SERVERS: kafka1:19092,kafka2:29092,kafka3:39092
    depends_on:
      - zookeeper-1
      - zookeeper-2
      - zookeeper-3
      
  kafka-3:
    image: confluentinc/cp-kafka:latest
    container_name: kafka-3
    ports:
      - '39092:39092'
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-3:9092,PLAINTEXT_HOST://localhost:39092
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      BOOTSTRAP_SERVERS: kafka1:19092,kafka2:29092,kafka3:39092
    depends_on:
      - zookeeper-1
      - zookeeper-2
      - zookeeper-3

now when I try to create topic with 3 partitions on it I get error:

INFO [Admin Manager on Broker 1]: Error processing create topic request   
CreatableTopic(name='__transaction_state', numPartitions=50, replicationFactor=3,  
assignments=[], configs=[CreateableTopicConfig(name='compression.type',  
value='uncompressed'), CreateableTopicConfig(name='cleanup.policy', value='compact'),  
CreateableTopicConfig(name='min.insync.replicas', value='2'),  
CreateableTopicConfig(name='segment.bytes', value='104857600'),  
CreateableTopicConfig(name='unclean.leader.election.enable', value='false')]) 
(kafka.server.ZkAdminManager)

kafka-1        | 
org.apache.kafka.common.errors.InvalidReplicationFactorException: 
Replication factor: 3 larger than available brokers: 2.

Why it sees only 2 brokers, not 3? And sometimes the error message indicates it sees only 1 broker.

mmuehlbeyer · 16 December 2021 06:25

Hi @programista4k

did you already check the docker logs of the kafka containers?

After a short view on your compose it seems that your advertised listener configuration is not correct
the port should differ for each container

should be

KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-1:9092,PLAINTEXT_HOST://localhost:19092

KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-2:9093,PLAINTEXT_HOST://localhost:29092

KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-3:9094,PLAINTEXT_HOST://localhost:39092

for reference there is a great blog post by @rmoff about advertised_listener

HTH,
Michael

programista4k · 22 December 2021 02:01

Yes Sir, I have read the Kafkas’ and Zookeepers’ logs and there is no error instead of Replication factor: 3 larger than available brokers: 2. when I try to create topic with 3 replicas.

I have fixed the ports as ya suggested. Now it looks like this and the problem is still alive:

version: '3.8'
services:

  zookeeper-1:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper-1
    ports:
      - '12181:2181'
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  zookeeper-2:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper-2
    ports:
      - '22181:2181'
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  zookeeper-3:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper-3
    ports:
      - '32181:2181'    
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      

  kafka-1:
    image: confluentinc/cp-kafka:latest
    container_name: kafka-1
    ports:
      - '19092:19092'
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-1:9092,PLAINTEXT_HOST://localhost:19092
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      BOOTSTRAP_SERVERS: kafka1:19092,kafka2:29092,kafka3:39092
      KAFKA_LOG_MESSAGE_TIMESTAMP_TYPE: LogAppendTime
    depends_on:
      - zookeeper-1
      - zookeeper-2
      - zookeeper-3

  kafka-2:
    image: confluentinc/cp-kafka:latest
    container_name: kafka-2
    ports:
      - '29092:29092'
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-2:9093,PLAINTEXT_HOST://localhost:29092
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      BOOTSTRAP_SERVERS: kafka1:19092,kafka2:29092,kafka3:39092
      KAFKA_LOG_MESSAGE_TIMESTAMP_TYPE: LogAppendTime
    depends_on:
      - zookeeper-1
      - zookeeper-2
      - zookeeper-3

  kafka-3:
    image: confluentinc/cp-kafka:latest
    container_name: kafka-3
    ports:
      - '39092:39092'
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-3:9094,PLAINTEXT_HOST://localhost:39092
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      BOOTSTRAP_SERVERS: kafka1:19092,kafka2:29092,kafka3:39092
      KAFKA_LOG_MESSAGE_TIMESTAMP_TYPE: LogAppendTime
    depends_on:
      - zookeeper-1
      - zookeeper-2
      - zookeeper-3

mmuehlbeyer · 22 December 2021 09:25

hey,
I’ve tested around with your docker-compose and detected some issues.
would you please try with the follwing docker-compose.yml

version: '3.8'
services:

  zookeeper-1:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper-1
    ports:
      - '2181:2181'
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  zookeeper-2:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper-2
    ports:
      - '2182:2182'
    environment:
      ZOOKEEPER_CLIENT_PORT: 2182
      ZOOKEEPER_TICK_TIME: 2000

  zookeeper-3:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper-3
    ports:
      - '2183:2183'
    environment:
      ZOOKEEPER_CLIENT_PORT: 2183
      ZOOKEEPER_TICK_TIME: 2000


  kafka-1:
    image: confluentinc/cp-kafka:latest
    container_name: kafka-1
    ports:
      - '19092:19092'
      - '9092:9092'
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_INTERNAL:PLAINTEXT,LISTENER_EXTERNAL:PLAINTEXT
      KAFKA_LISTENERS: LISTENER_INTERNAL://kafka-1:19092,LISTENER_EXTERNAL://kafka-1:9092
      KAFKA_ADVERTISED_LISTENERS: LISTENER_INTERNAL://kafka-1:19092,LISTENER_EXTERNAL://kafka-1:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_INTERNAL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_LOG_MESSAGE_TIMESTAMP_TYPE: LogAppendTime
    depends_on:
      - zookeeper-1
      - zookeeper-2
      - zookeeper-3

  kafka-2:
    image: confluentinc/cp-kafka:latest
    container_name: kafka-2
    ports:
      - '29092:29092'
      - '9093:9093'
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_INTERNAL:PLAINTEXT,LISTENER_EXTERNAL:PLAINTEXT
      KAFKA_LISTENERS: LISTENER_INTERNAL://kafka-2:29092,LISTENER_EXTERNAL://kafka-2:9093
      KAFKA_ADVERTISED_LISTENERS: LISTENER_INTERNAL://kafka-2:29092,LISTENER_EXTERNAL://kafka-2:9093
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_INTERNAL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_LOG_MESSAGE_TIMESTAMP_TYPE: LogAppendTime
    depends_on:
      - zookeeper-1
      - zookeeper-2
      - zookeeper-3

  kafka-3:
    image: confluentinc/cp-kafka:latest
    container_name: kafka-3
    ports:
      - '9094:9094'
      - '39092:39092'
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_INTERNAL:PLAINTEXT,LISTENER_EXTERNAL:PLAINTEXT
      KAFKA_LISTENERS: LISTENER_INTERNAL://kafka-3:39092,LISTENER_EXTERNAL://kafka-3:9094
      KAFKA_ADVERTISED_LISTENERS: LISTENER_INTERNAL://kafka-3:39092,LISTENER_EXTERNAL://kafka-3:9094
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_INTERNAL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_LOG_MESSAGE_TIMESTAMP_TYPE: LogAppendTime
    depends_on:
      - zookeeper-1
      - zookeeper-2
      - zookeeper-3

the following works (at least in my dev env)

kafka-topics --create --partitions 3 --replication-factor 3 --topic quickstart-events --bootstrap-server kafka-1:9092

programista4k · 22 December 2021 16:38

Thank you Sir.

programista4k · 22 December 2021 19:57

And what about those zookeepers, why only first zookeeper is used, Fren?
Shouldn’t it be:
zookeeper-1 for kafka-1,
zookeeper-2 for kafka-2
etc.?

or zookeper-1,zookeeper-2,zookeeper-3 for every kafka broker?

mmuehlbeyer · 23 December 2021 06:02

just for ease of configuration

you could also set each zookeeper for every kafka broker

programista4k · 24 December 2021 10:06

Tank you, Sir.

system · 31 December 2021 10:07

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
We need to implement a Kafka cluster with docker-compose. Facing Issue with Inter broker communication between machines. Can anyone help? Architecture and Design	3	2960	7 December 2022
Is my computer OK tu run kafka? I don't know what's wrong Kafka Streams	0	2159	31 January 2023
Load testing Kafka broker "Broker may not be available." Containers	0	2666	9 June 2023
My local Kafka instance stuck in "auto leader balancing" Kafka Streams	1	857	20 November 2023
Creating Kafka-connect with Azure-storage-sink in docker compose Kafka Connect	2	1518	17 August 2023

What is wrong with my cluster setup?

Related Topics