Corelating multiple events arriving in different topics

Kafka Streams
1

I am new to kafka, help is much appreciated
i have scenario where account creation and product order coming in different topic i want to co relate both orders and process it with dependency where account creation event get processed earlier than product order / contact create. kindly help how this can be achieved any link documentation or example would be great help

1 Like
2

I am also curious to find the solution to this. From what I have seen, the go-to solution is to use a single topic for ingestion.

The key to get this working is that you should probably partition by customer id into a single topic. This way, you can have “customer events” topic and events come out in order, by partition.

If you don’t have any way to correlate a series of events, then you would have to use one partition, which is just not acceptable for most cases.

The second answer on this post:
https://stackoverflow.com/questions/73105592/how-to-guarantee-message-ordering-over-multiple-topics-in-kafka

Talks about using union types, which is the norm for guaranteeing ordering.

The main drawbacks of union types for me are:

  1. hard to transition to them if you already have multiple topics set up
  2. could be hard to parse/validate which message you receive

Essentially, if you want ordering across multiple partitions/topics, you can’t! It makes sense when you understand that kafka splits work by topic/partition.

Here is tutorial about union types:

3

How do you want to correlate them? Are you just looking to filter orders down to those that happened after account creation? Check out Flink SQL interval joins to do this. The tutorial here looks for events in a shipments topic that occurred within 7 days of the corresponding order from an orders topic. You could massage this example to look for orders with event time occurring after the corresponding account creation event’s time.

1 Like
4

Hello,
According to me to ensure account creation is processed before product orders, use stream processing tools like Apache Kafka with Kafka Streams or Apache Flink . Create separate streams for account creation and product orders, then join them to correlate events.
Thanks