Use case:
Read/consume from another team’s /external Kafka cluster topic using Kafka Connect which is hosted on AKS cluster.
Question:
Can we use kafka connect without creating below internal topics in remote kafka cluster. if not, what is the alternative?
Please advise.
Hi,
not 100% sure about your scenario/use-case.
But based on your question I assume that you want to do the following:
“”“You don’t have any internal / own kafka cluster you just want to use a Kafka Sink Connector to get data out from another kafka cluster which is not under your control
“””
If this assumption is correct than I think that there is no way arround these topics if you want to use a distributed connect cluster. Kafka Connect needs topics to manage it’s own state and that is what those topics are used for.
In “stand-alone” mode it might be possible - but I never used that one.
With distributed mode I only see the possibility to get your own kafka cluster, setup mirrormaker2 to replicate the data from your source cluster to yours (if needed with less retention) and config a connector to get the data out of your cluster.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.