Can Kafka Stream made alert app on multiple topics dynamically?

Hi everyone,
I have a question,
I want develop an application that read and check condition of data record on Kafka topics, then if the condition match with an data record (or multiple with windows), then I rise alerts for that.

But i have a problem, When user register use app, user will register user’s topics for my application know it, each user has some topics.
How kafka-stream can handle it when the topics are dynamically ?
If all user use same topic I think it effect to performance of each users.

Thanks all.
Sorry for bad english.

Hi @ndhcoder ,

I think you have multiple options here:

  1. You could indeed use one topic for all users, partition by user, and start multiple Kafka Streams clients to scale out and improve performance. Make sure you set the number of partitions of the input topic high enough to be able to scale out sufficiently.
  2. You could enforce a pattern on the registered input topics and create a stream with a pattern (i.e., StreamsBuilder#stream(final Pattern topicPattern)). This is similar to 1, but I guess in your setup you would not have control of the number of partitions of the input topics which might limit scale out.
  3. You could create a topology for each registered user and create and start a KafkaStreams object for each topology. That basically boils down to having a distinct Kafka Streams application for each registered user. You could start each application in its own JVM or use one JVM for a given number of Kafka Streams applications.
  4. You could also have a mix of option 1 and 3. You have a Kafka Streams application for each registered user that only writes the records from the user specified input topics to a common input topic that is managed by you. Then, you apply option 1 on this input topic.

The best option depends on your use case and setup.

Best,
Bruno

2 Likes

Wow. Thanks you so much, Sir <3 <3 <3, I will try it now .

Hi Mr Bruno, In 2th solution,
I will define pattern for all topic, when new user register i create new topics follow the patterns, so on each topics I just need 1-2 partition and I can easy scale topic when new user register. It’s ok ?