Hi there!
According to @rmoff’s great blog post
It’s possible to reset source connectors by finding the connector-specific offset in the offsets topic defined by offset.storage.topic.
I’ve tested this out with a single task and it works great!
But what if you have multiple source connector tasks? Does each task create such a record?
Hi @helpermethod!
Off the top of my head, IIRC each connector will have an entry on the topic with a key based on the connector name plus other data specific to the connector (in the blog example it’s the name of the file being ingested; for JDBC connector I’d expect it to be the table name, etc).
So yes - I believe this method should work even with multiple connector tasks. I’d recommend testing it properly first though. If you bump up the log levels in the Kafka Connect worker you should be able to see enough detail of what’s being read from the topic to check things are behaving.
Hi @rmoff thanks for the hint, I’ll need to try this out.
I’ve condensed your blog post into a nice, native CLI tool (which uses Java 16, picocli and GraalVM Native Image).
It already works for the simple case, but I need to test it in more complex scenarios.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.