# Historic data load when joining (or aggregating) 2 streams

**URL:** <https://forum.confluent.io/t/historic-data-load-when-joining-or-aggregating-2-streams/2192>\
**Category:** Kafka Streams\
**Created:** [9 July 2021 16:30 UTC](https://forum.confluent.io/t/historic-data-load-when-joining-or-aggregating-2-streams/2192 "2021-07-09T16:30:14Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jasmith](https://avatars.discourse-cdn.com/v4/letter/j/cc9497/32.png) [@jasmith](https://forum.confluent.io/u/jasmith)\
**Post date:** [9 July 2021 16:30 UTC](https://forum.confluent.io/t/historic-data-load-when-joining-or-aggregating-2-streams/2192/1 "2021-07-09T16:30:14Z")

</div>

Say you have 2 streams - chat(keyed by chatId on the topic but rekeyed to userId for this join) & reaction-info(keyed by userId).

Whenever a new chat event is created I want to join to the latest reaction-info sent before the chat event. Users can have multiple chats & multiple reaction-infos

Can either do a left join to a reaction-info table  
Or reduce and compare timestamps of events to ensure we get a reaction info before the chat event.  
Or probably lots of other things.

However, I also have a db with historic data for these events and short of passing them in order I don’t know how I can ensure that I get the most up to date reaction info event occuring before each chat event. Guidance on an approach to this would be huuuugely appreciated.

I’d prefer to reduce on two streams to allow for corrective updates but would appreciate suggestions on how to load the history correctly with the reaction-info being a stream or a table.

Is there some handy settings that make this waaay easier to do on a table than it is to do in a reduce?

Cheers!

---

<div class="post-metadata">

**Author:** ![mjsax](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mjsax/32/3113_2.png) [@mjsax](https://forum.confluent.io/u/mjsax)\
**Post date:** [12 July 2021 21:23 UTC](https://forum.confluent.io/t/historic-data-load-when-joining-or-aggregating-2-streams/2192/2 "2021-07-12T21:23:31Z")

</div>

In general, Kafka Streams processes records in timestamp order. Thus, if your input data is properly timestamped, it should work out-of-the-box. Increasing the config `max.task.idle.ms` should help to improve the synchronization.

Cf [Temporal-Joins in Kafka Streams and ksqlDB | Matthias Sax, Confluent](https://www.confluent.io/events/kafka-summit-europe-2021/temporal-joins-in-kafka-streams-and-ksqldb/)

If you need to handle out-of-order data it’s more complex and it might be best to use the Processor API… We are working on improved out-of-order record handling for stream-table joins already, but it will take some time until we can ship it…

---

<div class="post-metadata">

**Author:** ![jasmith](https://avatars.discourse-cdn.com/v4/letter/j/cc9497/32.png) [@jasmith](https://forum.confluent.io/u/jasmith)\
**Post date:** [12 July 2021 21:41 UTC](https://forum.confluent.io/t/historic-data-load-when-joining-or-aggregating-2-streams/2192/3 "2021-07-12T21:41:45Z")

</div>

Thanks for the reply mjsax!

> [@mjsax](#):
>
> In general, Kafka Streams processes records in timestamp order.

Is this true even of globalktables? The table will be empty on startup so I’m presuming what you’ve said still applies. I’m in control of when the events get produced so I can certainly produce them in order with a larger `max.task.idle.ms` on the streams app.

Thanks again, temporal-joins sound fuuuuuuun!  
Jack

---

<div class="post-metadata">

**Author:** ![mjsax](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/mjsax/32/3113_2.png) [@mjsax](https://forum.confluent.io/u/mjsax)\
**Post date:** [12 July 2021 21:59 UTC](https://forum.confluent.io/t/historic-data-load-when-joining-or-aggregating-2-streams/2192/4 "2021-07-12T21:59:59Z")

</div>

> [@jasmith](#):
>
> Is this true even of globalktables?

No. `GlobalKTables` are not time-synchronized. Did not realize that you are using a `GlobalKTable`.

---

<div class="post-metadata">

**Author:** ![jasmith](https://avatars.discourse-cdn.com/v4/letter/j/cc9497/32.png) [@jasmith](https://forum.confluent.io/u/jasmith)\
**Post date:** [12 July 2021 22:12 UTC](https://forum.confluent.io/t/historic-data-load-when-joining-or-aggregating-2-streams/2192/5 "2021-07-12T22:12:33Z")

</div>

Okay cool,

I was just clarifying the exceptions to the rule in my own head.

Thanks again!