Lets say 30 producers are sending data to the same Kafka topic every 2 seconds at the same time. From my C# code I want to read cyclic (every 2 seconds) the messages only from the last 2 seconds. How can I do that? I tried a lot but couldn’t succeed. Could you share a very simple sample code?

You can accomplish this by getting the offsets corresponding to the desired timestamp (2 seconds ago) via OffsetsForTimes , and then seeking to those offsets: consumer.Subscribe(topic); ... var seekTimestamp = new Timestamp(DateTime.Now.Subtract(TimeSpan.FromSeconds(2))); var offsets = consumer.Of…

Hello I have completed the code, according to your answer as following. But I think I make something wrong. My goal is to read every two seconds the messages of the last two seconds. That would be in ideal case 30 messages (from 30 consumers that produce data at the same time to the same topic). Bu…

[image] mdarende: For every while loop I get one result This is expected - in each iteration you’re calling Consume which just gets one result. You’d need to keep going. There isn’t a single method you can call to get all within a time window. [image] mdarende: My goal is to read every t…

Hello Dave Thank you for the great suggestion. Sorry, as I am recently working with Kafka, I would ask how I could realise in this case the event based consuming. Could you give a very basic example for that? I mean how will I be informed of an event , so that I can just consume in this case. Than…

[image] mdarende: how will I be informed of an event , so that I can just consume in this case. Streaming apps often keep running, i.e., you would Consume in a loop. The consumer is always “on” and continuously consumes. In this case, there’s no need to separately be informed of an event and …

hello Dave I have explained my use case in the following topic: [image] Consuming data from Kafka topic for a leight weight Live-Chart app. in .NET Non-Java Clients Hello In my lightweigt C# .NET app I have to visualise the data, of a Kafka topic in a simple live chart.…

Read messages from the last x seconds of a topic

Clients

mdarende 20 March 2024 15:26 7

hello Dave

I have explained my use case in the following topic:

Topic		Replies	Views
I want to read just the last (newest) message in a Kafka topic Clients	2	18040	3 April 2023
How can I reset the topic offset while consuming from Kafka topic Non-Java Clients	2	2184	21 March 2024
Cyclic data reading from topic partitions takes too long (7-8 second) What could be the problem? Kafka Connect	0	1057	11 November 2023
Consuming data from Kafka topic for a leight weight Live-Chart app. in .NET Non-Java Clients	0	786	20 March 2024
How can I read the last message from a Kafka topic that has three partitions? Kafka Connect	0	1853	9 November 2023

Read messages from the last x seconds of a topic

Related topics