Containerized Connect listeners vs rest.advertised

mattimo · 3 June 2022 02:09

Im attempting to run a Kafka Connect container on Heroku, inside a private space.

Im hoping for help setting:
CONNECT_LISTENERS, CONNECT_REST_PORT, CONNECT_REST_ADVERTISED_HOST_NAME, CONNECT_REST_ADVERTISED_PORT.

From my understanding:
CONNECT_LISTENERS - this is the external endpoint for Kafka REST… thus CONNECT_LISTENERS="http://myapp.herokuapp.com:$PORT}

CONNECT_REST_PORT - If the process doesn’t bind to the servers port [Heroku dynos] within 60s, the dyno reboots. So CONNECT_REST_PORT=${PORT}

CONNECT_REST_ADVERTISED_LISTENER="HTTP" - This is http as heroku routes via http

Here comes the issue… Ive tried a variety of configs for the following…

CONNECT_REST_ADVERTISED_HOST_NAME - From the docs, this is the internal hostname/ip - so with (private space DNS discovery)[Private Spaces DNS Service Discovery | Heroku Dev Center], I can get an internal hostname of the form http://web.myapp.app.localspace: and with a private space, I can use a static port - say 5000.

The issue is, my REST API fails to bind…

2022-06-03T02:08:22.869326+00:00 app[web.1]: [2022-06-03 02:08:22,869] INFO jetty-9.4.43.v20210629; built: 2021-06-30T11:07:22.254Z; git: 526006ecfa3af7f1a27ef3a288e2bef7ea9dd7e8; jvm 11.0.13+8-LTS (org.eclipse.jetty.server.Server)
2022-06-03T02:08:22.894502+00:00 app[web.1]: [2022-06-03 02:08:22,891] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed)
2022-06-03T02:08:22.894503+00:00 app[web.1]: org.apache.kafka.connect.errors.ConnectException: Unable to initialize REST server
2022-06-03T02:08:22.894503+00:00 app[web.1]: at org.apache.kafka.connect.runtime.rest.RestServer.initializeServer(RestServer.java:216)
2022-06-03T02:08:22.894504+00:00 app[web.1]: at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:101)
2022-06-03T02:08:22.894504+00:00 app[web.1]: at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80)
2022-06-03T02:08:22.894505+00:00 app[web.1]: Caused by: java.io.IOException: Failed to bind to myapp.herokuapp.com/52.202.52.153:33635
2022-06-03T02:08:22.894505+00:00 app[web.1]: at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
2022-06-03T02:08:22.894506+00:00 app[web.1]: at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
2022-06-03T02:08:22.894506+00:00 app[web.1]: at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
2022-06-03T02:08:22.894507+00:00 app[web.1]: at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
2022-06-03T02:08:22.894507+00:00 app[web.1]: at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
2022-06-03T02:08:22.894507+00:00 app[web.1]: at org.eclipse.jetty.server.Server.doStart(Server.java:401)
2022-06-03T02:08:22.894508+00:00 app[web.1]: at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
2022-06-03T02:08:22.894508+00:00 app[web.1]: at org.apache.kafka.connect.runtime.rest.RestServer.initializeServer(RestServer.java:214)
2022-06-03T02:08:22.894509+00:00 app[web.1]: ... 2 more
2022-06-03T02:08:22.894509+00:00 app[web.1]: Caused by: java.net.BindException: Cannot assign requested address
2022-06-03T02:08:22.894510+00:00 app[web.1]: at java.base/sun.nio.ch.Net.bind0(Native Method)
2022-06-03T02:08:22.894510+00:00 app[web.1]: at java.base/sun.nio.ch.Net.bind(Net.java:455)
2022-06-03T02:08:22.894510+00:00 app[web.1]: at java.base/sun.nio.ch.Net.bind(Net.java:447)
2022-06-03T02:08:22.894511+00:00 app[web.1]: at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227)
2022-06-03T02:08:22.894511+00:00 app[web.1]: at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80)
2022-06-03T02:08:22.894512+00:00 app[web.1]: at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
2022-06-03T02:08:22.894512+00:00 app[web.1]: ... 9 more
2022-06-03T02:08:27.675168+00:00 heroku[web.1]: Error R10 (Boo

What’s the correct differentiation between CONNECT_LISTENERS and CONNECT_REST_ADVERTISED_HOST_NAME for managed instances? I can configure as I wish, my ssl handshake completes, I can ping my kafka instance… but I cannot for the life of me stand up kafka connect

mmuehlbeyer · 3 June 2022 06:39

hi @mattimo

welcome to the forum

CONNECT_LISTENERS:
defines the URIs the api will listen on
will to bind to all interfaces if set to 0.0.0.0

e.g. CONNECT_LISTENERS: 'http://0.0.0.0:8083'

CONNECT_REST_ADVERTISED_HOST_NAME
the hostname that will be given out to other workers to connect to.
should be set to resolvable hostname

hth,
michael

mattimo · 3 June 2022 22:37

Hi @mmuehlbeyer , thanks for the response. I know what these values should be from the confluent/apache docs, but my question is on deeper prod configuration…

Can I get your thoughts on this example config?
CONNECT_LISTENERS - List of external hostname/ip:port where Kafaka REST listens on… this would be the URI I use to get a list of connectors, post a new connector, delete a connector, etc… So this would be http://myapp.herokuapp.com:${PORT}…?

CONNECT_REST_ADVERTISED_HOST_NAME - internal hostname of the connect worker. Im in a Private Space and can use [Private Spaces DNS Service Discovery | Heroku Dev Center](https://DNS Service Discovery) to allow dyno-to-dyno communication [worker-to-worker].

After enabling this feature and binding my app to a static port (e.g. 5000) and HEROKU_PRIVATE_IP, my app is reachable via internal traffic at:
http://web.myapp.app.localspace:5000/

Should CONNECT_LISTENERS also contain the CONNECT_REST_ADVERTISED_HOST_NAME:CONNECT_REST_ADVERTISED_PORT`?
My app binds successfully with CONNECT_LISTENERS="HTTP://0.0.0.0:${PORT}, but all other combinations of external or internal hostname/ip:port I’ve tried cause my app to fail…

Can you provide a prod example here?

mattimo · 6 June 2022 21:47

If anyone finds this, I solved by taking advantage of static port available in private spaces:

CONNECT_REST_ADVERTISED_HOST_NAME="${HEROKU_DNS_DYNO_NAME}"
CONNECT_LISTENERS="HTTP://0.0.0.0:${PORT},HTTP://0.0.0.0:8000"
CONNECT_REST_PORT="${PORT}"
CONNECT_REST_ADVERTISED_PORT=8000

mmuehlbeyer · 7 June 2022 12:42

hey @mattimo

just for curiosity which docker image version did you use?

best,
michael

mattimo · 7 June 2022 19:28

@mmuehlbeyer I used confluentinc/cp-kafka-connect-base:6.1.4 and extended the image from there. I plan to post a public repo & walk-through soon - this was quite difficult to get up-and-running on Heroku, inside a Private Space… Happy to send the repo over once I do, I think it would benefit people who come across in the future… this was quite a headache to set up correctly (Heroku docs are very sparse)

Would you mind taking a look at my recent post Debezium SQLServer connector failed SSL handshake? I’m very close to finally setting this up…

system · 7 July 2022 19:29

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Possible typo in the kafka listeners configuration in rmoff's blog? Clients	2	3304	16 April 2021
Kafka listener and advertised.listener Ops	5	5739	7 September 2021
Setting advertised.listeners Containers	1	3359	16 May 2023
I want to duplicate Connect Server, but an error occurs Kafka Connect	2	1455	15 December 2023
Heroku Kafka Connect to snowflakes using docker file is not working Managed Connectors	3	3737	22 February 2021

Containerized Connect listeners vs rest.advertised

Related topics