I’m using trial version of kSQL hosted by Confluent Clould.
I’ve created a topic that I populate with K6 (50 messages/s).
Using GitHub - tomasfabian/ksqlDB.RestApi.Client-DotNet: ksqlDb.RestApi.Client is a C# LINQ-enabled client API for issuing and consuming ksqlDB push queries and executing statements. SqlServer.Connector is a client API for consuming row-level table changes (CDC - Change Data Capture) from Sql Server databases with the Debezium connector streaming platform. I’ve created a simple consumer in .NET Core 6
In kSQL I’ve created a Stream. When I consume it on .NET side the latency is in the ballpark of <100ms (as expected on my laptop) with no batches.
Push query (e.g. select * from table) on any Table that I create on the back of that stream with GROUP BY expression arrives to my consumer in 3s batches:
I’ve also tried various types/sizes of windows but 3s baches are consistent.
Is there anything I could do to reduce the batch duration to <1s?