Containerized Connect listeners vs rest.advertised

mattimo · 3 June 2022 02:09

Im attempting to run a Kafka Connect container on Heroku, inside a private space.

Im hoping for help setting:
CONNECT_LISTENERS, CONNECT_REST_PORT, CONNECT_REST_ADVERTISED_HOST_NAME, CONNECT_REST_ADVERTISED_PORT.

From my understanding:
CONNECT_LISTENERS - this is the external endpoint for Kafka REST… thus CONNECT_LISTENERS="http://myapp.herokuapp.com:$PORT}

CONNECT_REST_PORT - If the process doesn’t bind to the servers port [Heroku dynos] within 60s, the dyno reboots. So CONNECT_REST_PORT=${PORT}

CONNECT_REST_ADVERTISED_LISTENER="HTTP" - This is http as heroku routes via http

Here comes the issue… Ive tried a variety of configs for the following…

CONNECT_REST_ADVERTISED_HOST_NAME - From the docs, this is the internal hostname/ip - so with (private space DNS discovery)[Private Spaces DNS Service Discovery | Heroku Dev Center], I can get an internal hostname of the form http://web.myapp.app.localspace: and with a private space, I can use a static port - say 5000.

The issue is, my REST API fails to bind…

2022-06-03T02:08:22.869326+00:00 app[web.1]: [2022-06-03 02:08:22,869] INFO jetty-9.4.43.v20210629; built: 2021-06-30T11:07:22.254Z; git: 526006ecfa3af7f1a27ef3a288e2bef7ea9dd7e8; jvm 11.0.13+8-LTS (org.eclipse.jetty.server.Server)
2022-06-03T02:08:22.894502+00:00 app[web.1]: [2022-06-03 02:08:22,891] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed)
2022-06-03T02:08:22.894503+00:00 app[web.1]: org.apache.kafka.connect.errors.ConnectException: Unable to initialize REST server
2022-06-03T02:08:22.894503+00:00 app[web.1]: at org.apache.kafka.connect.runtime.rest.RestServer.initializeServer(RestServer.java:216)
2022-06-03T02:08:22.894504+00:00 app[web.1]: at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:101)
2022-06-03T02:08:22.894504+00:00 app[web.1]: at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80)
2022-06-03T02:08:22.894505+00:00 app[web.1]: Caused by: java.io.IOException: Failed to bind to myapp.herokuapp.com/52.202.52.153:33635
2022-06-03T02:08:22.894505+00:00 app[web.1]: at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
2022-06-03T02:08:22.894506+00:00 app[web.1]: at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
2022-06-03T02:08:22.894506+00:00 app[web.1]: at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
2022-06-03T02:08:22.894507+00:00 app[web.1]: at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
2022-06-03T02:08:22.894507+00:00 app[web.1]: at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
2022-06-03T02:08:22.894507+00:00 app[web.1]: at org.eclipse.jetty.server.Server.doStart(Server.java:401)
2022-06-03T02:08:22.894508+00:00 app[web.1]: at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
2022-06-03T02:08:22.894508+00:00 app[web.1]: at org.apache.kafka.connect.runtime.rest.RestServer.initializeServer(RestServer.java:214)
2022-06-03T02:08:22.894509+00:00 app[web.1]: ... 2 more
2022-06-03T02:08:22.894509+00:00 app[web.1]: Caused by: java.net.BindException: Cannot assign requested address
2022-06-03T02:08:22.894510+00:00 app[web.1]: at java.base/sun.nio.ch.Net.bind0(Native Method)
2022-06-03T02:08:22.894510+00:00 app[web.1]: at java.base/sun.nio.ch.Net.bind(Net.java:455)
2022-06-03T02:08:22.894510+00:00 app[web.1]: at java.base/sun.nio.ch.Net.bind(Net.java:447)
2022-06-03T02:08:22.894511+00:00 app[web.1]: at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227)
2022-06-03T02:08:22.894511+00:00 app[web.1]: at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80)
2022-06-03T02:08:22.894512+00:00 app[web.1]: at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
2022-06-03T02:08:22.894512+00:00 app[web.1]: ... 9 more
2022-06-03T02:08:27.675168+00:00 heroku[web.1]: Error R10 (Boo

What’s the correct differentiation between CONNECT_LISTENERS and CONNECT_REST_ADVERTISED_HOST_NAME for managed instances? I can configure as I wish, my ssl handshake completes, I can ping my kafka instance… but I cannot for the life of me stand up kafka connect

mmuehlbeyer · 3 June 2022 06:39

hi @mattimo

welcome to the forum

CONNECT_LISTENERS:
defines the URIs the api will listen on
will to bind to all interfaces if set to 0.0.0.0

e.g. CONNECT_LISTENERS: 'http://0.0.0.0:8083'

CONNECT_REST_ADVERTISED_HOST_NAME
the hostname that will be given out to other workers to connect to.
should be set to resolvable hostname

hth,
michael

mattimo · 3 June 2022 22:37

Hi @mmuehlbeyer , thanks for the response. I know what these values should be from the confluent/apache docs, but my question is on deeper prod configuration…

Can I get your thoughts on this example config?
CONNECT_LISTENERS - List of external hostname/ip:port where Kafaka REST listens on… this would be the URI I use to get a list of connectors, post a new connector, delete a connector, etc… So this would be http://myapp.herokuapp.com:${PORT}…?

CONNECT_REST_ADVERTISED_HOST_NAME - internal hostname of the connect worker. Im in a Private Space and can use [Private Spaces DNS Service Discovery | Heroku Dev Center](https://DNS Service Discovery) to allow dyno-to-dyno communication [worker-to-worker].

After enabling this feature and binding my app to a static port (e.g. 5000) and HEROKU_PRIVATE_IP, my app is reachable via internal traffic at:
http://web.myapp.app.localspace:5000/

Should CONNECT_LISTENERS also contain the CONNECT_REST_ADVERTISED_HOST_NAME:CONNECT_REST_ADVERTISED_PORT`?
My app binds successfully with CONNECT_LISTENERS="HTTP://0.0.0.0:${PORT}, but all other combinations of external or internal hostname/ip:port I’ve tried cause my app to fail…

Can you provide a prod example here?

mattimo · 6 June 2022 21:47

If anyone finds this, I solved by taking advantage of static port available in private spaces:

CONNECT_REST_ADVERTISED_HOST_NAME="${HEROKU_DNS_DYNO_NAME}"
CONNECT_LISTENERS="HTTP://0.0.0.0:${PORT},HTTP://0.0.0.0:8000"
CONNECT_REST_PORT="${PORT}"
CONNECT_REST_ADVERTISED_PORT=8000

mmuehlbeyer · 7 June 2022 12:42

hey @mattimo

just for curiosity which docker image version did you use?

best,
michael

mattimo · 7 June 2022 19:28

@mmuehlbeyer I used confluentinc/cp-kafka-connect-base:6.1.4 and extended the image from there. I plan to post a public repo & walk-through soon - this was quite difficult to get up-and-running on Heroku, inside a Private Space… Happy to send the repo over once I do, I think it would benefit people who come across in the future… this was quite a headache to set up correctly (Heroku docs are very sparse)

Would you mind taking a look at my recent post Debezium SQLServer connector failed SSL handshake? I’m very close to finally setting this up…

system · 7 July 2022 19:29

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
I want to duplicate Connect Server, but an error occurs Kafka Connect	2	928	15 December 2023
Confluent Http Sink & Azure Event Hubs Self-Managed Connectors	2	3450	2 August 2021
Java Code For Connects Kafka Connect	3	2950	27 May 2021
JDBC source connector for Oracle 11g is not working Kafka Connect	1	3046	25 August 2021
Kafka Connect distributed mode Dockerfile Kafka Connect	4	4183	13 May 2022

Containerized Connect listeners vs rest.advertised

Related Topics