Do multiple source connector tasks create multiple distinct connect offsets?

Hi there!

According to @rmoff’s great blog post

It’s possible to reset source connectors by finding the connector-specific offset in the offsets topic defined by offset.storage.topic.

I’ve tested this out with a single task and it works great!

But what if you have multiple source connector tasks? Does each task create such a record?

Hi @helpermethod!

Off the top of my head, IIRC each connector will have an entry on the topic with a key based on the connector name plus other data specific to the connector (in the blog example it’s the name of the file being ingested; for JDBC connector I’d expect it to be the table name, etc).

So yes - I believe this method should work even with multiple connector tasks. I’d recommend testing it properly first though. If you bump up the log levels in the Kafka Connect worker you should be able to see enough detail of what’s being read from the topic to check things are behaving.

2 Likes

Hi @rmoff thanks for the hint, I’ll need to try this out.

I’ve condensed your blog post into a nice, native CLI tool (which uses Java 16, picocli and GraalVM Native Image).

It already works for the simple case, but I need to test it in more complex scenarios.

1 Like