How to Create a Topic in Kafka after Firing it with Docker Compose?

Clients
1

This is how my docker compose file looks like.

services:
  broker:
    image: apache/kafka:latest
    container_name: broker
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_NUM_PARTITIONS: 3
    ports:
      - 9092:9092

So, kafka is starting up as expected, but I also want to create two topics called user and asset to capture events related to users and assets respectively. But I am unable to do it. Things I have tried

  • Docker exec inside the running container to find an appropriate shell command to do it. But the /bin/ directory does not have any binary or shell script to create topics. In any case, I find docker exec really hackish and a bad practice (unless for debugging) to avoid if I can.
  • Attach a command field to the yaml, but it does not help either, for the same reason.

So what would be the ways to create topics, best if within the docker compose itself?

2

The Apache Kafka command line tools live in /opt/kafka/bin/. So you could exec and run this to create a topic:

/opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 \
                               --create --topic new-topic

It’s possible to do this within the Docker Compose file. You could either have a custom image, or use command (doc) to override how the container starts. That requires knowledge of the image definition (here for the apache/kafka image), though, and would be subject to breaking in future versions. If you really want this all wrapped up in a Docker Compose file, I’d favor leaving the broker container alone and adding a separate tools container to do client stuff.