Hi all, strange, sure I had this working before from Golang and from Python, but having issue when connecting from Kafkacat…
Georges-MacBook-Pro.local:/Users/george/Desktop/ProjectsCommon/cp-all-in-one > docker-compose ps
Name Command State Ports
---------------------------------------------------------------------------------------------------------------------------------------------
broker /etc/confluent/docker/run Up 0.0.0.0:9092->9092/tcp,:::9092->9092/tcp, 0.0.0.0:9101->9101/tcp,:::9101->9101/tcp
connect /etc/confluent/docker/run Up 0.0.0.0:8083->8083/tcp,:::8083->8083/tcp, 9092/tcp
control-center /etc/confluent/docker/run Up 0.0.0.0:9021->9021/tcp,:::9021->9021/tcp
ksql-datagen bash -c echo Waiting for K ... Up
ksqldb-cli /bin/sh Up
ksqldb-server /etc/confluent/docker/run Up 0.0.0.0:8088->8088/tcp,:::8088->8088/tcp
rest-proxy /etc/confluent/docker/run Up 0.0.0.0:8082->8082/tcp,:::8082->8082/tcp
schema-registry /etc/confluent/docker/run Up 0.0.0.0:8081->8081/tcp,:::8081->8081/tcp
zookeeper /etc/confluent/docker/run Up 0.0.0.0:2181->2181/tcp,:::2181->2181/tcp, 2888/tcp, 3888/tcp
Georges-MacBook-Pro.local:/Users/george/Desktop/ProjectsCommon/cp-all-in-one >
AND
eorges-MacBook-Pro.local:/Users/george/Desktop/ProjectsCommon > docker run --rm -t --entrypoint /bin/sh edenhill/kafkacat:1.6.0 -c '
> # Install stuff
> apk add gpsd gpsd-clients
>
> nc 153.44.253.27 5631 | \
> gpsdecode | \
> kafkacat \
> -X bootstrap.servers=127.0.0.1:9092 \
> -t ais -P
> '
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/6) Installing libgcc (8.3.0-r0)
(2/6) Installing libstdc++ (8.3.0-r0)
(3/6) Installing gpsd (3.18.1-r2)
(4/6) Installing ncurses-terminfo-base (6.1_p20190518-r2)
(5/6) Installing ncurses-libs (6.1_p20190518-r2)
(6/6) Installing gpsd-clients (3.18.1-r2)
Executing busybox-1.30.1-r3.trigger
OK: 16 MiB in 31 packages
%3|1629273577.960|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 2ms in state CONNECT)
%3|1629273578.957|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
% ERROR: Local: Broker transport failure: 127.0.0.1:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 2ms in state CONNECT)
% ERROR: Local: All broker connections are down: 1/1 brokers are down : terminating
Georges-MacBook-Pro.local:/Users/george/Desktop/ProjectsCommon >
have to wonder… if I have a larger problem on my local cluster…
I can connect to environment using offset explorer…
but not using kafkacat (except if I got settings wrong), but I also can’t connect to the control-centre.
I’ve also tried (little different from above, also tried getting onto cluster via the zookeeper port: no luck.
… the docker command is thinking that localhost is internal to it…
need to some how point it to my host/external to the container - reintroduce Robin’s network block piece.
Will advise.
G
I’ve added a entry to my /etc/hosts file called broker1
hmmmm… I can ping my physical host from inside my docker container using this broker1 name.
then when trying to post events… I get:
/ # kafkacat -b broker1:9092 -t ais -P
{“uid”:6,“name”:“Cliff”,“locale”:“en_US”,“address_city”:“St Louis”,“elite”:“P”}
%3|1629286931.334|FAIL|rdkafka#producer-1| [thrd:localhost:9092/1]: localhost:9092/1: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1629286931.416|FAIL|rdkafka#producer-1| [thrd:localhost:9092/1]: localhost:9092/1: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
%3|16292
was part of Robin’s original Demo, ability to run it in a container in AWS, where it pulls the information from the publisher and then pushes to the confluent cluster.
I got that working in AWS, pushing to Confluent cluster.
Not getting it working when running same command on local system pushing to local cluster.
Interesting enough, got ksqldb cli working in container connecting to local cluster.
G
… stepping back, will just re read the blog and like the results, with ym current minimal execution knowledge, I’m finding there seems to be gaps in the script/code.
ie It talks about Reefers, creates a table, but no where is it linking up with the css file in the git repo how that gets into the table.
For someone that has enough experience it might be a great demo to try and execute
locally, (see there is a docker-compose.yml file that also includes Elastic and Kibana, but not seeing how/where the connectors are created to be able to ship data from the Kafka side to the Elastic side.
These blogs assume more knowledge that what I currently have, and battling to build this up between day job/expectations
G
I realise this post is a little old but I thought it would still be worth asking. Did you figure this out? I’m having the same issue where I can ping broker from within the kafkacat container, all the containers are on the same docker network, but when I try to run:
# nc 153.44.253.27 5631 | gpsdecode | kafkacat -b broker:9092 -t ais -P
%3|1710857032.098|FAIL|rdkafka#producer-1| [thrd:localhost:9092/1]: localhost:9092/1: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1710857032.218|FAIL|rdkafka#producer-1| [thrd:localhost:9092/1]: localhost:9092/1: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
% ERROR: Local: Broker transport failure: localhost:9092/1: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
% ERROR: Local: Broker transport failure: localhost:9092/1: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
It seems that the docker routing inside the kafkacat container is broken? I’m pretty familiar with docker but know nothing about kafka so this seems wrong but I’m not sure.
Just installed gpsdecode+kafkacat on my machine and reran the command from the host machine, pointing at localhost:9092; the docker mapped port, and it all works.
hi there
As allot of this is running inside containers… you need to be aware of the external ip of the service and address it as such, and when talking from localhost to container and vice versa… use actual ip addresses, not localhost, and not 127.0.0.1 as for the container thats the container itself.
In the compose file the broker container is given the hostname “broker” and can be pinged from the kafkacat container. The IP that is returned from the “ping broker” command shows that it’s on the docker network.
In the compose file there isn’t a docker network specified but I’m just running under the assumption that they all end up connected to the default docker network… (might be worth editing the compose file to have a specified network if that’s the issue).
Using the “kafkacat -b broker:9092 -t ais -P” command shows that it’s trying to get to “localhost:9092” rather than the address on the docker network. So it’s a little weird that the ping command, and kafkacat command seem to think that “broker” is at a different address:
#On Hostmachine
user@hostmachine: docker exec -it kafkacat sh
#Now inside container:
user@container: ping broker
PING broker (192.168.192.6): 56 data bytes
64 bytes from 192.168.192.6: seq=0 ttl=64 time=0.120 ms
64 bytes from 192.168.192.6: seq=1 ttl=64 time=0.124 ms
...
Aaaah! It’s an issue with the configuration of the broker! It believes itself to be “localhost” and so when asked for configuration info (bootstrap) then it responds with “the broker is at localhost:9092”. Here’s a nice blog post about it:
as hinted above… that 127.0.0.1 and localhost is a nasty one, it’s has different values in the container world depending on where you standing and viewing from and to.
i’ve learned to create local /etc/hosts entries for some of what i do and then use those, where possible.
G
