The use case I’m trying to implement reads events from a topic and processes them using KSQLDB into a final topic which is consumed with Kafka Connect into a REST API. Those events are in charge of notify users.
I’m trying to implement a new solution that can publish events with delay in the final topic. If the event needs to be delayed, it should be scheduled to a new timeframe. The logic is very easy using KSQLDB, resulting in two processed topics, one for events that should be notified instantly and the other one for events that should be scheduled, which contains the total seconds of delay. This is a simplified diagram of the solution:
The difficult part right now is choosing a good solution to implement this scheduler, because of Kafka inability to delay message sending. Af the moment I have two alternatives and I would like to know if they make sense to you:
1. Using another MQ message broker that allow scheduling, like ActiveMQ
This broker will only be used for the scheduled events, as a mechanism to add delay to the messages that needs it.
The way to do this should be simple, just use the Delay and Schedule Message Delivery functionality that AMQ provides and use a simple Java applications that reads from the scheduled event topic, publish into AMQ adding the delay and before that another process that reads from the AMQ and writes the events inside of the Instant event topic.
The bad thing about this solution is that we are deploying a new system just for managing a scheduler, and I’m not sure if it is the most efficient way to do this.
2. Using Kafka Streams State Store and Punctuators
The idea comes from this presentation when they show a way to use Kafka Streams Punctuators to pull events at different periods of time.
This solution is nice because it uses all native Kafka applications and don’t rely on external systems, but the complexity of developing and maintaining it is higher than the first one and also it is not truly real time because the punctuators rely on a time scheduler and programmed processes, so they don’t trigger an action when a new event arrives.
What do you think about this two options? Do you know another alternatives using Kafka Streams and maybe a simple scheduler inside the application? Every help is very welcome!
Thanks a lot for your help,
Brais