Kafka Stream Help

Kafka Streams
1

Hello,

I have already written simple Kafka Streams jobs and now I have a slightly more complicated one. I have googled quite a bit but am not sure if I need a statestore or a materiallized view.
Most importantly, before I always created a new message again for each message. But now I want to create a new message based on many input messages.

I have several signals S1,S2,S3, etc. these have three attributes each.
SignalName(String), Timestamp, Value(boolean).

Each signal is a stream that has the value “false” for a time and then becomes “true” for a certain time. I want to determine the time the signal is true.

The new message should look like this.
New message:
SignalName(String), StartTrue(Timestamp), EndTrue(Timestamp.)

Input:

S1 T1 false
S1 T2 true
S1 T3 true
S1 T4 false
S1 T5 false
S1 T6 true
S1 T7 true
S1 T8 true
S1 T9 false

Output:

S1 T2 T3
S1 T6 T8

I hope I could explain the use case.
Does anyone have an example or tips I can follow?

Thanks.

2

If you program you would use a state store. A materialized view is just a higher lever concept/abstractions on top of a state store.

For you problem: you can use a Processor (or Transformer if you use the DSL) and attach a state store to it. If a new message comes in, use the signal field as key (maybe you would need to repartition the data if not partitioned by signal already) and do a lookup into the state store. If not found, and the signal is off (ie, false), just drop the message. If not found and the signal is true, put key/timestamp into the state store. If found, and the signal is still true, drop the message. If found, and the signal is false, compute the duration as difference between the current message timestamp and the timestamp in the store, delete the entry from the state store, and send the result message downstream.