Read/consume from Remote Kafka cluster

Kafka Connect
1

Use case:

Read/consume from another team’s /external Kafka cluster topic using Kafka Connect which is hosted on AKS cluster.

Question:

Can we use kafka connect without creating below internal topics in remote kafka cluster. if not, what is the alternative?

  1. config.storage.topic
  2. offset.storage.topic
  3. status.storage.topic

Please advise.

2

Hi,

not 100% sure about your scenario/use-case.
But based on your question I assume that you want to do the following:
“”“You don’t have any internal / own kafka cluster you just want to use a Kafka Sink Connector to get data out from another kafka cluster which is not under your control
“””

If this assumption is correct than I think that there is no way arround these topics if you want to use a distributed connect cluster. Kafka Connect needs topics to manage it’s own state and that is what those topics are used for.

In “stand-alone” mode it might be possible - but I never used that one.

With distributed mode I only see the possibility to get your own kafka cluster, setup mirrormaker2 to replicate the data from your source cluster to yours (if needed with less retention) and config a connector to get the data out of your cluster.

1 Like
3

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.