Even I am facing same kind of issue with ksql, I am not able to pull/select queries based on time. Does Kafka sql support that kind of feature or no?
Can someone please help/respond…, was waiting to see if this actually works or not.!
Can you be more specific about what query you’re running, what errors/warnings you see in the ksqlDB server log, what data is in the topic, what version of the software you’re running, etc?
My question is similar to this : KSQL Query to consume after certain time from insertion
I am looking exactly for the same…
ksqldb-server:0.18.0 , ksqldb-cli:0.18.0…
There is no error on ksql cli, but I am not able to see any data in my required query after half and hour.
My actually requirement is that some clients will publish records/messages to my ABC topic where functionality is as follows I need to consume and wait for half an hour and then Insert them into postgress sql DB. So I thought of using Ksql in order to remove the steps of files/jobs/storage so that the message/records can stay in topic for 30 mins and can be inserted directly into postgress DB.
Can you give some more context around why you need to do this? Why do you need this delay? You’re going to be going against the grain of a streaming world by introducing artificial constraints in at this point of the flow
Once we get the record for third party client they need some time approx 30 mins to activate in their systems. So, we want to wait for 30 mins before storing it into postgress Database to display it in UI properly without any missing information.
That’s the reason I thought of blocking the record in Kafka Topic for 30 mins rather than putting them in external storage services (In KSQL, my query should pick the records after 30 mins from their arrival time into topic) .
Why not just use a predicate in the UI when it queries Postgres to filter the data?
Otherwise you’re building into your pipeline an artificial constraint that, if your 3rd party were to reduce their latency would leave you with further work to do to rectify.
Yes I agree, our functionality is bit diff. If we have such constraints in UI it will be an additional task and impacts the performance because the condition will be applied every time the user logs in to the application. And not only at UI we are using this data in many others places and it is really not a good idea to put conditions where ever it being used. So in order to eliminate all of these bottlenecks and make the application to work in optimised way, I was thinking to put waiting time before inserting into postgress db.
Just wanted to get confirmation if ksql supports this kind of behaviour !
Robin, can you please respond ? Based on your answer about ksql, if it doesn’t support the required way we will think about other ways around.