Hi All,
I am a new bee to the Kafka streaming data.
Like to reach out to the Kafka community to get some feedback.
It is possible to have Metadata driven data quality framework for the streaming data?
What are the different approach we have to implement the Data Quality to the streaming data.
I haven’t found much in blog or community discussion around data quality framework. Seems like we can query the DQ rules in KSQL but it is hardcoded value in the query instead we want centralized metadata table to maintain the DQ rules and reusability.
I am not 100% certain what you are trying to accomplish. However, I spoke to the PM for ksqlDB and they suggested you may want dynamic persistent queries:
You can parameterize queries, and when you want to change their values you just bounce the server running it - or restart the query.