Connecting to broker - docker-compose

Hi all, strange, sure I had this working before from Golang and from Python, but having issue when connecting from Kafkacat…

Georges-MacBook-Pro.local:/Users/george/Desktop/ProjectsCommon/cp-all-in-one > docker-compose ps
     Name                    Command               State                                         Ports
---------------------------------------------------------------------------------------------------------------------------------------------
broker            /etc/confluent/docker/run        Up      0.0.0.0:9092->9092/tcp,:::9092->9092/tcp, 0.0.0.0:9101->9101/tcp,:::9101->9101/tcp
connect           /etc/confluent/docker/run        Up      0.0.0.0:8083->8083/tcp,:::8083->8083/tcp, 9092/tcp
control-center    /etc/confluent/docker/run        Up      0.0.0.0:9021->9021/tcp,:::9021->9021/tcp
ksql-datagen      bash -c echo Waiting for K ...   Up
ksqldb-cli        /bin/sh                          Up
ksqldb-server     /etc/confluent/docker/run        Up      0.0.0.0:8088->8088/tcp,:::8088->8088/tcp
rest-proxy        /etc/confluent/docker/run        Up      0.0.0.0:8082->8082/tcp,:::8082->8082/tcp
schema-registry   /etc/confluent/docker/run        Up      0.0.0.0:8081->8081/tcp,:::8081->8081/tcp
zookeeper         /etc/confluent/docker/run        Up      0.0.0.0:2181->2181/tcp,:::2181->2181/tcp, 2888/tcp, 3888/tcp
Georges-MacBook-Pro.local:/Users/george/Desktop/ProjectsCommon/cp-all-in-one >

AND

eorges-MacBook-Pro.local:/Users/george/Desktop/ProjectsCommon > docker run --rm -t --entrypoint /bin/sh edenhill/kafkacat:1.6.0 -c '
> # Install stuff
> apk add gpsd gpsd-clients
>
> nc 153.44.253.27 5631 | \
>   gpsdecode | \
>   kafkacat \
>   -X bootstrap.servers=127.0.0.1:9092 \
>     -t ais -P
> '
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/6) Installing libgcc (8.3.0-r0)
(2/6) Installing libstdc++ (8.3.0-r0)
(3/6) Installing gpsd (3.18.1-r2)
(4/6) Installing ncurses-terminfo-base (6.1_p20190518-r2)
(5/6) Installing ncurses-libs (6.1_p20190518-r2)
(6/6) Installing gpsd-clients (3.18.1-r2)
Executing busybox-1.30.1-r3.trigger
OK: 16 MiB in 31 packages
%3|1629273577.960|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 2ms in state CONNECT)
%3|1629273578.957|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
% ERROR: Local: Broker transport failure: 127.0.0.1:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 2ms in state CONNECT)
% ERROR: Local: All broker connections are down: 1/1 brokers are down : terminating
Georges-MacBook-Pro.local:/Users/george/Desktop/ProjectsCommon >

Note to self on the docker-compose problem, re the removing of images…/service. Think I just figured it out,

docker-compose up -d pulls image,
docker-compose down stops and removes it.

docker-compose stop just stops it, and leaves there, ready again for
docker-compose start

user error in the end, been using docker-compose down instead of docker-compose stop…
G

have to wonder… if I have a larger problem on my local cluster…
I can connect to environment using offset explorer…
but not using kafkacat (except if I got settings wrong), but I also can’t connect to the control-centre.

I’ve also tried (little different from above, also tried getting onto cluster via the zookeeper port: no luck.

docker run --rm -t --entrypoint /bin/sh edenhill/kafkacat:1.6.0 -c '
# Install stuff
apk add gpsd gpsd-clients 

nc 153.44.253.27 5631 | \
  gpsdecode | \
  kafkacat -b localhost:9092  -t ais -P
'

… More diagnostics… Strange… this worked.

Georges-MacBook-Pro.local:/Users/george > kafkacat -b localhost:9092 -t ais -P
{“uid”:1,“name”:“Cliff”,“locale”:“en_US”,“address_city”:“St Louis”,“elite”:“P”}

{“uid”:2,“name”:“Cliff”,“locale”:“en_US”,“address_city”:“St Louis”,“elite”:“P”}
{“uid”:3,“name”:“Cliff”,“locale”:“en_US”,“address_city”:“St Louis”,“elite”:“P”}
{“uid”:4,“name”:“Cliff”,“locale”:“en_US”,“address_city”:“St Louis”,“elite”:“P”}
{“uid”:5,“name”:“Cliff”,“locale”:“en_US”,“address_city”:“St Louis”,“elite”:“P”}

I could see messages on topic via offset explorer, so thinking is something to do with above dockerization of the command.

G

ping ping ping…

… the docker command is thinking that localhost is internal to it…
need to some how point it to my host/external to the container - reintroduce Robin’s network block piece.
Will advise.
G

I’ve added a entry to my /etc/hosts file called broker1
hmmmm… I can ping my physical host from inside my docker container using this broker1 name.
then when trying to post events… I get:
/ # kafkacat -b broker1:9092 -t ais -P

{“uid”:6,“name”:“Cliff”,“locale”:“en_US”,“address_city”:“St Louis”,“elite”:“P”}
%3|1629286931.334|FAIL|rdkafka#producer-1| [thrd:localhost:9092/1]: localhost:9092/1: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1629286931.416|FAIL|rdkafka#producer-1| [thrd:localhost:9092/1]: localhost:9092/1: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
%3|16292

If I run kafkacat outside container all works.

Anyone see my error ?

G

one question?

why would you run kafkacat from within the container?

was part of Robin’s original Demo, ability to run it in a container in AWS, where it pulls the information from the publisher and then pushes to the confluent cluster.

I got that working in AWS, pushing to Confluent cluster.

Not getting it working when running same command on local system pushing to local cluster.

Interesting enough, got ksqldb cli working in container connecting to local cluster.
G

… stepping back, will just re read the blog and like the results, with ym current minimal execution knowledge, I’m finding there seems to be gaps in the script/code.
ie It talks about Reefers, creates a table, but no where is it linking up with the css file in the git repo how that gets into the table.
For someone that has enough experience it might be a great demo to try and execute
locally, (see there is a docker-compose.yml file that also includes Elastic and Kibana, but not seeing how/where the connectors are created to be able to ship data from the Kafka side to the Elastic side.

These blogs assume more knowledge that what I currently have, and battling to build this up between day job/expectations :frowning:
G