guys,
for a local confluent via docker-compose deployment (development),
have anyone done it using multiple brokers ? would love a copy of the docker compose file if I may.
Also curious howI would then “expose” my multiple brokers to my client.
G
below is my current docker-compose section as related to the kafka cluster
I want to change this to a 3 node/broker config
services:
# begin Confluent Kafka cluster
broker:
image: confluentinc/cp-kafka:7.9.1
container_name: broker
hostname: broker
ports:
- 9092:9092
- 9101:9101
- 29092:29092
environment:
KAFKA_NODE_ID: 1001
KAFKA_LISTENERS: 'PLAINTEXT://broker:29092,CONTROLLER://broker:29093,PLAINTEXT_HOST://0.0.0.0:9092'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_JMX_PORT: 9101
KAFKA_JMX_HOSTNAME: localhost
KAFKA_PROCESS_ROLES: 'broker,controller'
KAFKA_CONTROLLER_QUORUM_VOTERS: '1001@broker:29093'
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
KAFKA_LOG_DIRS: /var/lib/kafka/data
# Replace CLUSTER_ID with a unique base64 UUID using "bin/kafka-storage.sh random-uuid"
# See https://docs.confluent.io/kafka/operations-tools/kafka-tools.html#kafka-storage-sh
CLUSTER_ID: ${CLUSTER_ID}
volumes:
- ./data/confluent.d/broker/data:/var/lib/kafka/data
- ./data/confluent.d/broker/logs:/var/log/kafka
schema-registry:
image: confluentinc/cp-schema-registry:7.9.1
container_name: schema-registry
hostname: schema-registry
depends_on:
- broker
ports:
- 9081:9081
environment:
SCHEMA_REGISTRY_HOST_NAME: schema-registry
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: 'broker:29092'
SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:9081
control-center:
image: confluentinc/cp-enterprise-control-center:7.9.1
container_name: control-center
hostname: control-center
depends_on:
- broker
ports:
- 9021:9021 # -> Web UI console
environment:
CONTROL_CENTER_BOOTSTRAP_SERVERS: 'broker:29092'
CONTROL_CENTER_CONNECT_CONNECT-DEFAULT_CLUSTER: 'connect:8083'
CONTROL_CENTER_CONNECT_HEALTHCHECK_ENDPOINT: '/connectors'
CONTROL_CENTER_KSQL_KSQLDB1_URL: "http://ksqldb-server:8088"
CONTROL_CENTER_KSQL_KSQLDB1_ADVERTISED_URL: "http://localhost:8088"
CONTROL_CENTER_SCHEMA_REGISTRY_URL: "http://schema-registry:9081"
CONTROL_CENTER_REPLICATION_FACTOR: 1
CONTROL_CENTER_INTERNAL_TOPICS_PARTITIONS: 1
CONTROL_CENTER_MONITORING_INTERCEPTOR_TOPIC_PARTITIONS: 1
CONFLUENT_METRICS_TOPIC_REPLICATION: 1
PORT: 9021
# end Clonfluent cluster
Hi there
Looks good, I will try.
some questions though…
where we are not exposing a port externally, why are we listing it under ports: and incrementing an internal and external number.
ie:
broker01:
ports:
- 9092:9092
- 9101:9101
- 29092:29092
environment:
KAFKA_LISTENERS: ‘PLAINTEXT://broker01:29092,CONTROLLER://broker01:29193,PLAINTEXT_HOST://0.0.0.0:9092’
KAFKA_ADVERTISED_LISTENERS: ‘PLAINTEXT://broker01:29092,PLAINTEXT_HOST://localhost:9092’
broker02:
ports:
- 9093:9093
- 9102:9102
- 29093:29093
environment:
KAFKA_LISTENERS: ‘PLAINTEXT://broker02:29093,CONTROLLER://broker02:29194,PLAINTEXT_HOST://0.0.0.0:9093’
KAFKA_ADVERTISED_LISTENERS: ‘PLAINTEXT://broker02:29093,PLAINTEXT_HOST://localhost:9093’
why not just use 29092. they private on the host.
I understand ie 9092 that we expose to the localhost and where we expose to 0.0.0.0, so for boker01 we use localhost:9092 and broker host we use localhost:9093 etc.
G
Hi there
Ok, this does not seem to have worked, If I create 3 topics, one through each of the brokers only the broker to which the control center points reflects the topic.
If I say query each broker about what topics it knows about it only shows the topic created using it, so not acting like a 3 node/broker cluster.
G
I found the following…
wish this guy did one consolidated example.
curious. only ever deployed control center, never a controller. ???
would love if the guy followed more structure on his port numbering. sorry mildly OCD.
G
how did you create the topics?
docker compose exec broker01 kafka-topics \
--create -topic factory_iot_north \
--bootstrap-server localhost:9092 \
--partitions 1 \
--replication-factor 1 \
--config retention.ms=1000000
docker compose exec broker02 kafka-topics \
--create -topic factory_iot_south \
--bootstrap-server localhost:9093 \
--partitions 1 \
--replication-factor 1 \
--config retention.ms=1000000
docker compose exec broker03 kafka-topics \
--create -topic factory_iot_east \
--bootstrap-server localhost:9094 \
--partitions 1 \
--replication-factor 1 \
--config retention.ms=1000000
# Lets list topics, excluding the default Confluent Platform topics
docker compose exec broker03 kafka-topics \
--bootstrap-server localhost:9094 \
--list | grep -v '_confluent' |grep -v '__' |grep -v '_schemas' | grep -v 'default' | grep -v 'docker-connect'
I created them on purpose one via each broker to confirm they can all see all
if you use a replication factor of 1 the topic will reside on the broker you’ve created the topic on only
think i got it working, agree on the topic will only reside on one node, but all brokers must still know about all topics to be able to redirect traffic to who ever owns that topic from a leader perspective.
KAFKA_CONTROLLER_QUORUM_VOTERS: "1001@broker01:29193,1002@broker02:29194,1003@broker03:29195"
ye that def worked…
I changed the topics to have replication factor of 2, create succeeded, also even though I’m connected to control center via the first node, it’s telling the KRAFT Leader is 1002 which is the 2nd node.
for reference
services:
# begin Confluent Kafka cluster
broker01:
image: confluentinc/cp-kafka:7.9.1
container_name: broker01
hostname: broker01
ports:
- 9092:9092
- 9101:9101
- 29092:29092
environment:
KAFKA_NODE_ID: 1001
KAFKA_LISTENERS: 'PLAINTEXT://broker01:29092,CONTROLLER://broker01:29193,PLAINTEXT_HOST://0.0.0.0:9092'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker01:29092,PLAINTEXT_HOST://localhost:9092'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_JMX_PORT: 9101
KAFKA_JMX_HOSTNAME: localhost
KAFKA_PROCESS_ROLES: 'broker,controller'
KAFKA_CONTROLLER_QUORUM_VOTERS: "1001@broker01:29193,1002@broker02:29194,1003@broker03:29195"
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
KAFKA_LOG_DIRS: /var/lib/kafka/data
# Replace CLUSTER_ID with a unique base64 UUID using "bin/kafka-storage.sh random-uuid"
# See https://docs.confluent.io/kafka/operations-tools/kafka-tools.html#kafka-storage-sh
CLUSTER_ID: ${CLUSTER_ID}
volumes:
- ./data/confluent.d/broker01/data:/var/lib/kafka/data
- ./data/confluent.d/broker01/logs:/var/log/kafka
broker02:
image: confluentinc/cp-kafka:7.9.1
container_name: broker02
hostname: broker02
depends_on:
- broker01
ports:
- 9093:9093
- 9102:9102
- 29093:29093
environment:
KAFKA_NODE_ID: 1002
KAFKA_LISTENERS: 'PLAINTEXT://broker02:29093,CONTROLLER://broker02:29194,PLAINTEXT_HOST://0.0.0.0:9093'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker02:29093,PLAINTEXT_HOST://localhost:9093'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_JMX_PORT: 9102
KAFKA_JMX_HOSTNAME: localhost
KAFKA_PROCESS_ROLES: 'broker,controller'
KAFKA_CONTROLLER_QUORUM_VOTERS: "1001@broker01:29193,1002@broker02:29194,1003@broker03:29195"
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
KAFKA_LOG_DIRS: /var/lib/kafka/data
CLUSTER_ID: ${CLUSTER_ID}
volumes:
- ./data/confluent.d/broker02/data:/var/lib/kafka/data
- ./data/confluent.d/broker02/logs:/var/log/kafka
broker03:
image: confluentinc/cp-kafka:7.9.1
container_name: broker03
hostname: broker03
depends_on:
- broker01
- broker02
ports:
- 9094:9094
- 9103:9103
- 29094:29094
environment:
KAFKA_NODE_ID: 1003
KAFKA_LISTENERS: 'PLAINTEXT://broker03:29094,CONTROLLER://broker03:29195,PLAINTEXT_HOST://0.0.0.0:9094'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker03:29094,PLAINTEXT_HOST://localhost:9094'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_JMX_PORT: 9102
KAFKA_JMX_HOSTNAME: localhost
KAFKA_PROCESS_ROLES: 'broker,controller'
KAFKA_CONTROLLER_QUORUM_VOTERS: "1001@broker01:29193,1002@broker02:29194,1003@broker03:29195"
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
KAFKA_LOG_DIRS: /var/lib/kafka/data
CLUSTER_ID: ${CLUSTER_ID}
volumes:
- ./data/confluent.d/broker03/data:/var/lib/kafka/data
- ./data/confluent.d/broker03/logs:/var/log/kafka
schema-registry:
image: confluentinc/cp-schema-registry:7.9.1
container_name: schema-registry
hostname: schema-registry
depends_on:
- broker01
- broker02
- broker03
ports:
- 9081:9081
environment:
SCHEMA_REGISTRY_HOST_NAME: schema-registry
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: 'broker01:29092'
SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:9081
control-center:
image: confluentinc/cp-enterprise-control-center:7.9.1
container_name: control-center
hostname: control-center
depends_on:
- broker01
- broker02
- broker03
ports:
- 9021:9021 # -> Web UI console
environment:
CONTROL_CENTER_BOOTSTRAP_SERVERS: 'broker01:29092'
CONTROL_CENTER_CONNECT_CONNECT-DEFAULT_CLUSTER: 'connect:8083'
CONTROL_CENTER_CONNECT_HEALTHCHECK_ENDPOINT: '/connectors'
CONTROL_CENTER_KSQL_KSQLDB1_URL: "http://ksqldb-server:8088"
CONTROL_CENTER_KSQL_KSQLDB1_ADVERTISED_URL: "http://localhost:8088"
CONTROL_CENTER_SCHEMA_REGISTRY_URL: "http://schema-registry:9081"
CONTROL_CENTER_REPLICATION_FACTOR: 1
CONTROL_CENTER_INTERNAL_TOPICS_PARTITIONS: 1
CONTROL_CENTER_MONITORING_INTERCEPTOR_TOPIC_PARTITIONS: 1
CONFLUENT_METRICS_TOPIC_REPLICATION: 1
PORT: 9021
# end Clonfluent cluster
ok, one step further, the broker0#:2909X
Well as these ports are specific to the broker#, and note used externally they can all be the same, so cleaned up config to the next, also not exposing 2909# anymore to the host (port forward) as it’s not needed
services:
# begin Confluent Kafka cluster
broker01:
image: confluentinc/cp-kafka:7.9.1
container_name: broker01
hostname: broker01
ports:
- 9092:9092
- 9101:9101
environment:
KAFKA_NODE_ID: 1001
KAFKA_LISTENERS: 'PLAINTEXT://broker01:29092,CONTROLLER://broker01:29192,PLAINTEXT_HOST://0.0.0.0:9092'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker01:29092,PLAINTEXT_HOST://localhost:9092'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_JMX_PORT: 9101
KAFKA_JMX_HOSTNAME: localhost
KAFKA_PROCESS_ROLES: 'broker,controller'
KAFKA_CONTROLLER_QUORUM_VOTERS: "1001@broker01:29192,1002@broker02:29192,1003@broker03:29192"
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
KAFKA_LOG_DIRS: /var/lib/kafka/data
# Replace CLUSTER_ID with a unique base64 UUID using "bin/kafka-storage.sh random-uuid"
# See https://docs.confluent.io/kafka/operations-tools/kafka-tools.html#kafka-storage-sh
CLUSTER_ID: ${CLUSTER_ID}
volumes:
- ./data/confluent.d/broker01/data:/var/lib/kafka/data
- ./data/confluent.d/broker01/logs:/var/log/kafka
broker02:
image: confluentinc/cp-kafka:7.9.1
container_name: broker02
hostname: broker02
depends_on:
- broker01
ports:
- 9093:9093
- 9102:9102
environment:
KAFKA_NODE_ID: 1002
KAFKA_LISTENERS: 'PLAINTEXT://broker02:29092,CONTROLLER://broker02:29192,PLAINTEXT_HOST://0.0.0.0:9093'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker02:29092,PLAINTEXT_HOST://localhost:9093'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_JMX_PORT: 9102
KAFKA_JMX_HOSTNAME: localhost
KAFKA_PROCESS_ROLES: 'broker,controller'
KAFKA_CONTROLLER_QUORUM_VOTERS: "1001@broker01:29192,1002@broker02:29192,1003@broker03:29192"
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
KAFKA_LOG_DIRS: /var/lib/kafka/data
CLUSTER_ID: ${CLUSTER_ID}
volumes:
- ./data/confluent.d/broker02/data:/var/lib/kafka/data
- ./data/confluent.d/broker02/logs:/var/log/kafka
broker03:
image: confluentinc/cp-kafka:7.9.1
container_name: broker03
hostname: broker03
depends_on:
- broker01
- broker02
ports:
- 9094:9094
- 9103:9103
environment:
KAFKA_NODE_ID: 1003
KAFKA_LISTENERS: 'PLAINTEXT://broker03:29092,CONTROLLER://broker03:29192,PLAINTEXT_HOST://0.0.0.0:9094'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker03:29092,PLAINTEXT_HOST://localhost:9094'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_JMX_PORT: 9102
KAFKA_JMX_HOSTNAME: localhost
KAFKA_PROCESS_ROLES: 'broker,controller'
KAFKA_CONTROLLER_QUORUM_VOTERS: "1001@broker01:29192,1002@broker02:29192,1003@broker03:29192"
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
KAFKA_LOG_DIRS: /var/lib/kafka/data
CLUSTER_ID: ${CLUSTER_ID}
volumes:
- ./data/confluent.d/broker03/data:/var/lib/kafka/data
- ./data/confluent.d/broker03/logs:/var/log/kafka
schema-registry:
image: confluentinc/cp-schema-registry:7.9.1
container_name: schema-registry
hostname: schema-registry
depends_on:
- broker01
- broker02
- broker03
ports:
- 9081:9081
environment:
SCHEMA_REGISTRY_HOST_NAME: schema-registry
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: 'broker01:29092'
SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:9081
control-center:
image: confluentinc/cp-enterprise-control-center:7.9.1
container_name: control-center
hostname: control-center
depends_on:
- broker01
- broker02
- broker03
ports:
- 9021:9021 # -> Web UI console
environment:
CONTROL_CENTER_BOOTSTRAP_SERVERS: 'broker01:29092'
CONTROL_CENTER_CONNECT_CONNECT-DEFAULT_CLUSTER: 'connect:8083'
CONTROL_CENTER_CONNECT_HEALTHCHECK_ENDPOINT: '/connectors'
CONTROL_CENTER_KSQL_KSQLDB1_URL: "http://ksqldb-server:8088"
CONTROL_CENTER_KSQL_KSQLDB1_ADVERTISED_URL: "http://localhost:8088"
CONTROL_CENTER_SCHEMA_REGISTRY_URL: "http://schema-registry:9081"
CONTROL_CENTER_REPLICATION_FACTOR: 1
CONTROL_CENTER_INTERNAL_TOPICS_PARTITIONS: 1
CONTROL_CENTER_MONITORING_INTERCEPTOR_TOPIC_PARTITIONS: 1
CONFLUENT_METRICS_TOPIC_REPLICATION: 1
PORT: 9021
# end Clonfluent cluster
Have to write this up 
1 Like
fixed some stuff in the gist also
I normally tend to use also different internal ports to be sure on which broker I end up
but in the end it’s up to you 
we all have our habits…
G