Can standalone connector and distributed connector co-exist

Kafka Connect
1

HI all, we have a vendor application where there are some firewall limitation and I can only run that connector in a standalone host, where our broker already have a distributed connector running on some other hosts.

I found that when I start the standalone host connector in standalone-mode, it still try to run at the hosts that running the distributed connector, not sure how can I tell my standalone host connector run at the current host?

2

Hi @carfield

could you please provide some more details?

my understanding sofar:
host a: broker, distributed connector
host b: standalone

correct?

best,
michael

3

Can you elaborate on how you’re running this standalone connector? It sounds like perhaps you’re invoking it wrong, or using the wrong config file.

4

Thanks Michael, we are running at MSK env, it setup is like:

MSK: broker
HostA: distributed connector
HostB: distributed connector
HostC: standalone

where if check

HTTP://hostC:8083/connector/name/status

it say it running at HostA

5

thanks

as @rmoff could you provide some details how run the connector?

best,
michael

6

If you’re attempting to run multiple independent Connect systems on top of a single Kafka cluster, then you’ll need to adjust some configurations.

Connect clusters utilize a set of ‘internal topics’ and consumer group settings to share state and coordinate behavior. If you configure two clusters in such a way that they share that state, they will attempt to behave as cluster.

https://docs.confluent.io/home/connect/self-managed/userguide.html#kconnect-internal-topics

That said, i’m not certain from your question if your intention is to run the standalone worker within the existing distributed cluster or not, so you may need to provide more details as others have requested.

7

Thanks, Rick

My intention is that the standalone worker is separate from the existing distributed cluster. In fact, after I restart the processes of the existing cluster, the new standalone connector will not try to run at the cluster, which is good. However, somehow the standalone connector has some communication problem with the broker, and “poll()” is not calling, but that is a different question.

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.