Replaying time series data

Community Lounge
1

Hi everyone,
i use kafka in my ETL pipeline. and some time i have to replay data. my data is time series data which has one field as timestamp when it was produced,
is it possible, i can replay it, to reprocess from some specific point in time.

2

Hi @junedknit , what client library are you using? LibrdKafka based implementations have a function available to get offsets from a timestamp, which you can use to replay from said offsets. rd_kafka_offsets_for_times is the base function. librdkafka/rdkafka.h at 7230a1aa327b55ace54579d019e9484af96886c6 · edenhill/librdkafka · GitHub

1 Like
3

I’d echo what @Charlla said - client libraries usually support the ability to consume from a given offset. If you can clarify which client or consumer you’re using we can help better.