Golang Code not connecting to topic when containerised

[georgelza]

… so having been able to compile 1/2 problem…

Please see below:

FYI: little lab app…
Part A reads text file and publishes messages onto a topic.
Part B (the below) consumers from topic and push via gRPC/Protobuf to a “poster”
Part C “Poster” receives messages and does some more work, and eventually sinks messages into a Cassandra or PostgreSQL DB.

… so my code works fine if executed from a go file that I simply execute as go run consumer.go
problem, when i package it via a dockerfile and push into a minikube environment it gives the below errors. I get these, trying to connect to a CP stack deployed using the cp_all_in_one deployment using docker-compose and similarly get it if I do the local zip file with s"confluent local services start"

I’ve tried to point my client/consumer to the localhost from inside the pod/container and also to a physical ip.

INFO: 2021/04/21 09:06:22 Retrieving variables ..
INFO: 2021/04/21 09:06:22 Hostname is scrubber-55fdcfcc8-xmbsc
INFO: 2021/04/21 09:06:22 gRPC Server is localhost
INFO: 2021/04/21 09:06:22 gRPC Port is 9010
INFO: 2021/04/21 09:06:22 Kafka Broker is 192.168.49.2
INFO: 2021/04/21 09:06:22 Kafka Port is 9092
INFO: 2021/04/21 09:06:22 Kafka Topic is people_pb
INFO: 2021/04/21 09:06:22 Kafka # Parts is 3
INFO: 2021/04/21 09:06:22 Kafka Rep Factor is 1
INFO: 2021/04/21 09:06:22 Kafka Retension is 3600000
INFO: 2021/04/21 09:06:22 Kafka Group is tmg1
INFO: 2021/04/21 09:06:22 Debug Level is 2
INFO: 2021/04/21 09:06:22
INFO: 2021/04/21 09:06:22 **** Configure Admin Kafka Connection ****
INFO: 2021/04/21 09:06:22 acm_str is 192.168.49.2:9092
%3|1618995982.047|FAIL|rdkafka#producer-1| [thrd:192.168.49.2:9092/bootstrap]: 192.168.49.2:9092/bootstrap: Connect to ipv4#192.168.49.2:9092 failed: Connection refused (after 0ms in state CONNECT)
INFO: 2021/04/21 09:06:22 Admin client created
INFO: 2021/04/21 09:06:22 Context object created
INFO: 2021/04/21 09:06:22 ParseDuration Configured
INFO: 2021/04/21 09:06:22 [retention.ms](http://retention.ms/) Configured
%3|1618995983.049|FAIL|rdkafka#producer-1| [thrd:192.168.49.2:9092/bootstrap]: 192.168.49.2:9092/bootstrap: Connect to ipv4#192.168.49.2:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
%6|1618996011.041|FAIL|rdkafka#producer-1| [thrd:192.168.49.2:9092/bootstrap]: 192.168.49.2:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 3ms in state APIVERSION_QUERY)

anyone with ideas ?

if anyone have a go file + accompanying dockerfile that they know works, would love it, then I can confirm it’s not code but a setting, which I’m sure it is.

[georgelza]

… can add having the same errors when I simply do a docker build and a docker run (other words not using minikube)

INFO: 2021/04/21 09:47:46
INFO: 2021/04/21 09:47:46 **** Configure Admin Kafka Connection ****
INFO: 2021/04/21 09:47:46 acm_str is		localhost:9092
%3|1618998466.037|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
INFO: 2021/04/21 09:47:46 Admin client created
INFO: 2021/04/21 09:47:46 Context object created
INFO: 2021/04/21 09:47:46 ParseDuration Configured
INFO: 2021/04/21 09:47:46 retention.ms Configured
%3|1618998467.037|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)

G

[rmoff]

Hey George,

Have a look at this: Why Can’t I Connect to Kafka? | Troubleshoot Connectivity and also Learning Golang (some rough notes) - S02E08 - Checking Kafka advertised.listeners with Go

[georgelza]

thanks, will do, hopefully I’m just being dumb, not uncommon

will report back.

G

[georgelza]

dumb question, but where is server.properties so that I can change the advertised.listener setting.

G

[georgelza]

found it…

/etc/kafka/

G

[georgelza]

Sweeeeeet, Mr Robin is my hero, again, thanks. that resolved that issue, hehehehe

now for me onto to the next one, having I think a similar problem between the docker running the Part 2 / scrubber program and the Part 3 Poster, but think I just need to find the correct K8S service name to utilise.

G