Connect single ksqlDB to multiple Kafka clusters

ksqlDB
1

Is it possible to connect single ksqlDB cluster to multiple Kafka clusters ?

If in a hybrid setup, data originates in on-prem Kafka cluster C1 topic T1 and some data pipeline transforms, processes and pushes it to different Kafka cluster C2 topic T2 on cloud. Is it possible for one single ksqlDB cluster K1 to connect to both C1 and C2 kafka clusters to read data from T1 and T2, join them, and do streaming queries over the joined data?

Two old threads in this forum suggest that is not possible in KafkaStreams, implying ksqlDB may not be able to do this, but still checking whether newer versions have this support yet?

Thanks in advance

2

No, that’s not possible.

As you already figured out, Kafka Streams does not support it, and ksqlDB inherits this limitation. I would assume that Kafka Streams will support cross-cluster processing in the future, because this feature request is coming up regularly. There is no timeline for it though right now. And even after Kafka Streams supports it, ksqlDB would need to adopt this new feature afterwards.

3

This topic was automatically closed after 30 days. New replies are no longer allowed.