Hi all,
I’m looking for some guidance on an issue I’m having with my hopping windows.
I have a pipeline set up which accepts a nested json message into a stream which pushes messages into a hopping window if certain criteria are met.
The hopping window will then aggregate these messages and produce an alert onto a topic if the calculations within the windows hit defined thresholds.
The issue I am having is that individual messages are being included in multiple alerts when they shouldn’t be. For example, if I have it set up so 3 messages of a certain type are required to trigger an alert then I would want these 3 messages to aggregate in the window, trigger an alert and then be forgotten so that the next alert includes 3 completely new messages. In reality the same messages are included in different alerts.
Once an alert is triggered by a window the output message includes an array of IDs with each ID belonging to a message which led to the alert firing.
To attempt to fix my issue. I have exploded this array and created a table of ‘SEEN IDS’ which I am then left joining back into the hopping window. I am then using the FILTER function to remove messages which are already present in the SEEN IDs table.
In small volumes, this works well and I can see that messages are successfully filtered out in the hopping window and I am only getting unique messages contributing to my alerts. However, as soon as I increase the volume e.g. multiple messages per second the logic fails and the window is producing alerts before the IDs are being filtered so I am back to getting duplicate messages in different alerts.
I tried to get round this by introducing another ID tracker to catch those which slip through the net of the hopping window. So I would count how many times an ID has been seen if it is produced on to my alert topic and filter the messages again before they are produced onto my final topic. However, I get the same issue, as soon as I increase the message volume it fails.
Any advice on how to get around this would be much appreciated. Due to the type of calculations I am doing, the window has to be a hopping window.