Scheduling events in an Event-driven architecture

The use case I’m trying to implement reads events from a topic and processes them using KSQLDB into a final topic which is consumed with Kafka Connect into a REST API. Those events are in charge of notify users.

I’m trying to implement a new solution that can publish events with delay in the final topic. If the event needs to be delayed, it should be scheduled to a new timeframe. The logic is very easy using KSQLDB, resulting in two processed topics, one for events that should be notified instantly and the other one for events that should be scheduled, which contains the total seconds of delay. This is a simplified diagram of the solution:

The difficult part right now is choosing a good solution to implement this scheduler, because of Kafka inability to delay message sending. Af the moment I have two alternatives and I would like to know if they make sense to you:

1. Using another MQ message broker that allow scheduling, like ActiveMQ

This broker will only be used for the scheduled events, as a mechanism to add delay to the messages that needs it.

The way to do this should be simple, just use the Delay and Schedule Message Delivery functionality that AMQ provides and use a simple Java applications that reads from the scheduled event topic, publish into AMQ adding the delay and before that another process that reads from the AMQ and writes the events inside of the Instant event topic.

The bad thing about this solution is that we are deploying a new system just for managing a scheduler, and I’m not sure if it is the most efficient way to do this.

2. Using Kafka Streams State Store and Punctuators

The idea comes from this presentation when they show a way to use Kafka Streams Punctuators to pull events at different periods of time.

This solution is nice because it uses all native Kafka applications and don’t rely on external systems, but the complexity of developing and maintaining it is higher than the first one and also it is not truly real time because the punctuators rely on a time scheduler and programmed processes, so they don’t trigger an action when a new event arrives.

What do you think about this two options? Do you know another alternatives using Kafka Streams and maybe a simple scheduler inside the application? Every help is very welcome!

Thanks a lot for your help,

If you are using Spring Boot JonRunr might be nice for this. Just to setup, and can use almost anything for persistence. You even get a dashboard with all the jobs. You can read more here.

I did scheduling before in the way you described at 2).
I was no “huge scale” use-case we just had to schedule a few thousand (for sure < 100k) events at the same time. There it worked quite well and I think it also does also scale if you can horizontally scale the kstreams nodes ( and # partitions).

In the implementation we used this Kafka Stream Usage Patterns - Apache Kafka - Apache Software Foundation ("How to purge data from KTables based on age
") for inspiration. In this example the same pattern is used to delete entries from a KTable. In addition of deleting the entries from the scheduler state-store we also forward it to the target topic.