Consuming a Topic with .NET from the beginning

Sunspoticus · 5 January 2022 23:24

First off, I’m extremely new to Kafka but am an experienced .net developer. Using the .net examples, I’m able to connect and subscribe to a topic. Now I might very well be missing something in the documentation, but I can’t seem to find out how to get all the messages/data from beginning of time. Our organization publishes data to this kafka topic and I need to connect at least once to level-set the data before taking on deltas of it.

Any help would seriously be appreciated, thanks!

rmoff · 6 January 2022 11:08

Hi, welcome to the forum!

You need to set auto.offset.reset to earliest, which I believe in .NET is done in the ConsumerConfig with AutoOffsetReset

There’s an example here:

github.com

confluentinc/confluent-kafka-dotnet/blob/4de84f7c3c4b0d140f643dcb21110fd523efa2cc/examples/Consumer/Program.cs#L46


      
          /// </summary>
          public static void Run_Consume(string brokerList, List<string> topics, CancellationToken cancellationToken)
          {
              var config = new ConsumerConfig
              {
                  BootstrapServers = brokerList,
                  GroupId = "csharp-consumer",
                  EnableAutoCommit = false,
                  StatisticsIntervalMs = 5000,
                  SessionTimeoutMs = 6000,
                  AutoOffsetReset = AutoOffsetReset.Earliest,
                  EnablePartitionEof = true,
                  // A good introduction to the CooperativeSticky assignor and incremental rebalancing:
                  // https://www.confluent.io/blog/cooperative-rebalancing-in-kafka-streams-consumer-ksqldb/
                  PartitionAssignmentStrategy = PartitionAssignmentStrategy.CooperativeSticky
              };
          
          
    const int commitPeriod = 5;
          
          
    // Note: If a key or value deserializer is not set (as is the case below), the 
              // deserializer corresponding to the appropriate type from Confluent.Kafka.Deserializers

system · 18 January 2022 13:17

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
How can I reset the topic offset while consuming from Kafka topic Non-Java Clients	2	245	21 March 2024
Consuming data from Kafka topic for a leight weight Live-Chart app. in .NET Non-Java Clients	0	201	20 March 2024
I want to read just the last (newest) message in a Kafka topic Clients	2	4664	3 April 2023
How to Handle Kafka message offsets while using with Entity Framework Core Exception Confluent Cloud	0	2530	18 May 2022
Consuming messages in .NET client issues Non-Java Clients	0	3236	16 February 2022

Consuming a Topic with .NET from the beginning

Related Topics