Ksqldb aggregate window send only final message

ksqlDB
1

I have topic with collect view for users.

How i can in ksqldb send only final message for windows aggregate on close 10 minute?

Example after first 10 minute we have only 2 messages then windows close (center column on screen). “myApp” receive only 2 messages per user.

p.s. beautiful range of time is more for clarity

image

2

Try using EMIT FINAL instead of EMIT CHANGES – you should set an appropriate grace-period though (otherwise it default to 24h). Not though: EMIT FINAL uses an in-memory buffer and there is no guard against out-of-memory errors.

2 Likes
3
  1. i cant find nothing about EMIT FINAL in documentation
  2. is emit final persist or full memory ? example if after 5 min power off and after 1 min power on will continue processing or lose data from 5 min and start from zero ?
  3. what is emit final collect in memory buffer in my example? will be collect LIST of all raw message in window range or only result for every user with increment ‘viewCount’ ?
4

i cant find nothing about EMIT FINAL in documentation

It’s not documented… Because of the issue that you could hit an out-of-memory error, we don’t document it – and usually advice to use it only with care…

is emit final persist or full memory ? example if after 5 min power off and after 1 min power on will continue processing or lose data from 5 min and start from zero ?

It’s persistent. – But note, that it applies to fail-over or server shutdown only. If you terminate a query, state will be cleaned up. There is no such thing as “pausing” a query in ksqlDB.

what is emit final collect in memory buffer in my example? will be collect LIST of all raw message in window range or only result for every user with increment ‘viewCount’ ?

Both the aggregation and EMIT final only store the aggregation result (not the raw input data).

1 Like
5

Thanks! you answered my questions, opened my eyes how it works!
I love kafka and ksqldb !

  1. Another clarifying question why did you focus on grace-period?
  2. Are there any plans for the development of emit final and its inclusion in the official documentation?
1 Like
6

Another clarifying question why did you focus on grace-period?

Not sure what you mean by this?

Are there any plans for the development of emit final and its inclusion in the official documentation?

Yes. The goal is to fix the memory issue. After we have an implementation that is safe, we will document the feature.

7

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.