Read from Kafka cluster and write to Azure Event Hubs

Ops Cluster Replication
1

Use case:

  1. Read/consume from another team’s /external kafka cluster topic using Kafka Connect which is installed on AKS cluster that we manage.

  2. Write the messages to azure event Hub .

I am new in Kafka domain. Would like to get some guidance on how this can be achieved #1 & #2.

Thanks
Sujoy

2

Hi Sujoy,

It sounds like you’re describing MirrorMaker 2. This runs on Kafka Connect and consumes from one Kafka cluster and writes to another. I found this article which looks like it describes what you’re trying to do: Use Apache Kafka MirrorMaker - Azure Event Hubs - Azure Event Hubs | Microsoft Docs

3

Hi Robin - Thanks for the information. We are checking that option.
However, is there any other way to handle this use case just using out-of-box Kafka connect to connect two Kafka clusters (Source and Target) without MirrorMaker?

4

This is what MirrorMaker is. It’s literally out of the box.
Kafka Connect is the integration/execution framework, and MirrorMaker is the connector that implements it – both are part of Apache Kafka.

1 Like
5

Thanks Robin. got it.
On another note, since we are using confluent Kafka connect, can we use confluent replicator as well to replicate data from Source Kafka cluster to Target cluster which is Azure Event Hub ? basically, just trying to capture possible options here before we finalize the approach.

6

Hi @sujoy3456 , as per this documentation Replicator Overview | Confluent Documentation Confluent Replicator needs "The Origin and destination clusters to be Apache Kafka® or Confluent Platform. "

The reason is that Azure Event Hub is another implementation of the Kafka Protocol by Azure. While it has been proven that Replicator could work with Event Hub, this is not a tested nor supported setup

Have a good day

1 Like
7

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.