Window of message batch

Stream Processing ksqlDB
1

Hi there,

We can limit a stream for aggregation or stream-stream operations by window of time. Can we apply window of count of messages ?
For example, I want to sum of cost every for every 500 records in real time.

Is it possible to provide batch.size as we provide in consumer config in ksqlDB?

Thanks.

2

ksqlDB does not support count-size windows – only time-size windows.

Maybe you can use an UDF to implement what you need?

Is it possible to provide batch.size as we provide in consumer config in ksqlDB?

Yes, but it will only impact performance, not semantics. (cf ksqlDB Server Configuration Parameter Reference - ksqlDB Documentation – you can set any client config this way)

3

Thanks for your response.

I am getting sales data in JSON format and I want to create array of every 500 messages for one of its fields.

This is what I want to achieve. Yes UDF can help me here, As per my understanding UDF needs to be programmed and converted into jar to add into plugin path if KsqlDB.

But I am using confluent cloud so where shall I put UDF’s jar to leverage it?

4

UDFs are not supported in Confluent Cloud right now :frowning:

5

So do I left with option of setting up self managed ksqldb pointing to confluent cloud and using UDF on that self managed ksqldb ?

6

Yes, that is for sure an option.