Question for folks running Kafka in prod: how do you handle replaying/reprocessing a bounded slice of a topic? Typical case for us: “a consumer had a bug from Tue 14:00 to Wed 09:00 — re-send exactly those records to the retry topic.”
We started with ad-hoc scripts and they kept failing in the same ways: lost their position when they died mid-run, no record afterwards of what was actually replayed, and they needed shell access + credentials someone had to hand out every time. We ended up building a small internal service that runs replays as jobs — flexible start (offset / timestamp / group position), flexible end (time / offset / count), resumable after a restart, with an audit trail of what was re-sent where.
Genuinely curious how other teams do this at scale:
- Do you reset consumer-group offsets and re-consume, or copy records into a topic?
- What happens when it’s 2M records and the process dies halfway?
- Can anyone later see that a replay happened and exactly what was replayed?
- Has anyone built something similar in-house, or is there a tool you already use for this?
Trying to figure out whether we over-engineered this or whether every team quietly has the same thing sitting in a repo somewhere.
Two notes since you already posted to Slack #general: posting the same question to the Forum is fine and per plan — they’re separate surfaces and the Forum post is the durable one. But if the Slack thread starts generating real replies today, prioritize engaging there first; the Forum post can absorb attention over weeks, the Slack one can’t.