100DaysKafka, Linux, Docker, Windows 10

Community 100 Days of Code
1

I’m beginning the 100DaysKafka tutorial. Before I get too far down this path I want to confirm a few things:

  1. I do my development in Windows using IntelliJ, not Linux.
  2. Confluent requires Docker and Linux.
    Is it possible to run the Docker Confluent Kafka image from Intellij on Windows or should I just fight the battle with the broken Windows version of Kafka?
1 Like
2

Hi,

I would recommend using a Kafka in Docker Containers on Windows (and also on Mac or Linux).

On my private Win10 machine I have DockerDesktop installed (still free for personal use) and run the docker containers in WSL2. (I recommend using WSL2 with Docker if it is possible on your machine - for more details what is needed for this see: Get started with Docker containers on WSL | Microsoft Docs)

For my kafka stuff I usually have a docker-compose.yaml file describing the cluster + config I need in the corresponding project.
There I just can use “docker-compose up” and “docker-compose down” via Terminal in Intellij (or in WSl2 Shell) to start or stop the whole cluster.
Checkout the official docker-compose files from confluent: cp-all-in-one/cp-all-in-one at 7.0.0-post · confluentinc/cp-all-in-one · GitHub
Usually I try to reduce the needed stuff to a minimum (so e.g. remove ksql containers if I don’t need ksql right now, reduce zookeeper and broker to one instance) to reduce resource usage.

In my experience this is the easiest way to develop things with kafka locally.

Using Docker also has the advantage that you also can easily add other stuff like mongodb , elastic, postgres or whatever you need e.g. to explore Kafka Connect.

One common pitfall with that setup is the advertised hostname in the docker compose file

KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092

→ If here you don’t have this “localhost” listener configured you need to enter “broker” (or whatever hostname you use in the listener configured) to your local hosts file mapping to 127.0.0.1 otherwise the clients won’t be able to connect to cluster.
(But since confluent added the localhost listener there this shouldn’t be a problem anymore - my older version of this file doesn’t have this listener)

For me it was much easier to go the Docker way instead of getting everything running on Windows in a native way. If Docker is also complete new to you this might be an additional challenge but I think that it is worth to tackle this challenge.
(If you want it even more challenging you could also use minikube to run a local k8s and deploy your kafka with one of the available operators - but I think that is an overkill for your use-case ^^)

Instead of working with Kafka Locally you could also use confluent cloud. As I remember correctly there is some free usage you get when you sign up. And I remember that in the Confluent Podcasts they always promote Confluent Cloud with some additional Promo Code to get an extra 100$ of free usage. (But I don’t know how deep this 100DaysKafka goes into Kafka itself and if you can do that all with confluent cloud as there are still some limitations)

1 Like
3

Can I do the Java development part of this (which is what interests me as opposed to the configuration stuff) with IntelliJ on Windows 10? I do NOT do development in a Linux environment. Thanks!

4

Hi, yes.

Docker Desktop acts as a proxy between Docker containers (in WSL2) and Windows and you will be able to connect to the kafka cluster like you run it on windows.

5

Any examples of doing this IntelliJ on Windows to Docker to Linux to Kafka?

6

Thanks! I’ve found some pointers. Have the cp-all-in-one-community build running in Docker and open in Intellij. Next I want to install and run the Twitter app and experiment with it. Not sure how it will connect with the Docker process. Thanks again.

7

As the cp-all-in-one-community/docker-compose.yml contains:

    ports:
      - "29092:29092"
      - "9092:9092"
      - "9101:9101"

AND

      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092

You should be able to connect to the kafka cluster with “localhost:9092” in you bootstrap server config within your java app

Basically Docker maps the given Ports from the containers to WSL AND DockerDesktop makes them available directly in windows too.

8

I decided to go with the standard Confluent Cloud to start with. Have the first example Java application running with output in console! Thanks for your help :slight_smile:

1 Like