Hello,
Currently I am developing an interactive queries application. I’m sending messages (coming from pageviews on our website) through a pipeline with a machine learning scoring and store the results back into a topic.
This topic is fed back to a Kafka Streams application with a simple persistent key value store. It is used to enable REST calls to get the score for specific values (via session_id as key).
What I would like to know: How can I monitor the time when the score is finally set into the key value store and it is ready to be queried? The input message has the original time of the input message, I would like to monitor the difference from that time to the time it is stored in the store.
First of all, using ksql I can track the time difference of the scoring to the orginal event (a pageview on our website), which is around 2-3 seconds.
My first thoughts were:
- Monitor the time until it is in the changelog topic. This seems to be quite big (between 3 and 30 seconds)
- add a logging step in the kafka streams application when a score
- write the difference between current time when storing the value (in the kafka streams interactive query application) and original message time back to a new topic
Is there a common way to do such a thing? How would you do it?
Thx in advance!