Unknown Topic Or Partition error, even though the topic should be created automatically

Clients
1

Hi there! This will basically be a duplication of this issue I opened on kafka-gos GitHub-page. I noticed the issue first using the Golang-client, however the longer I research this and try stuff out, the more I’m thinking, that it might be related to Kafka (or the docker-images that I’m using) and not the client or kafkacat.

This is my first post to this forum, so please bear with me, should I not comply to any forum-standards or have overlocked something very dumb.

So here we go:

Describe the bug

When publishing a message, I get Unknown Topic Or Partition: the request is for a topic or partition that does not exist on this broker, event though it should create the message by default, as far as I understand the README.md.

It worked before, when I was using the exact same setup on NixOS (until I switched to Arch Linux yesterday).

Kafka Version

Latest docker-image from confluentinc (should be 2.7.1 if it’s the latest official release from their GitHub, as of today)

To Reproduce

Start Kafka using the following docker-compose.yaml:

version: "3.3"
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 22181:2181
  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 29092:29092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

execute the following commands:

docker-compose -f "FILE UNDER WHICH YOU'VE STORED THE ABOVE" up -d
mkdir tmp
cd tmp
go mod init example.com/tmp
go get -u github.com/segmentio/kafka-go
vim main.go

Fill main.go with the following lines:

package main

import (
	"context"

	"github.com/segmentio/kafka-go"
)

func main() {
	w := &kafka.Writer{
		Addr:  kafka.TCP("localhost:29092"),
		Topic: "hello",
	}

	err := w.WriteMessages(context.Background(), kafka.Message{
		Key:   []byte("hello"),
		Value: []byte("hello!"),
	})

	if err != nil {
		panic(err)
	}
}

Run main.go

go run main.go

The output was the following for me:

panic: [3] Unknown Topic Or Partition: the request is for a topic or partition that does not exist on this broker

goroutine 1 [running]:
main.main()
        /home/tim/tmp/main.go:21 +0x265
exit status 2

Expected behavior

The message should be published, even though the topic doesn’t exist yet.

Additional context

❯ uname -a
Linux t14s 5.12.10-arch1-1 #1 SMP PREEMPT Thu, 10 Jun 2021 16:34:50 +0000 x86_64 GNU/Linux

I installed Kubuntu 21.04 to try out if the issue also exists on an Ubuntu-system. Turns out: Yes, still getting the same errors as above after applying the same commands.

❯ uname -a
Linux t14s 5.11.0-18-generic #19-Ubuntu SMP Fri May 7 14:22:03 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

I also played around with the versions of the kafka- and zookeeper-images used in the docker-compose.yaml. Relaxing the versions to 5.4.4 doesn’t change the behavior.

Note, that I have similar issues, when I use kafkacat and try to subscribe to a topic, that doesn’t exist yet