Hi, I find that if we update the version of the connector, then offset will be reset and we will need to initialize it again, not sure if there are way to keep offset available for difference version of connector? We only store a sequence number there and there is not version difference
Source connector or sink connector?
Version of the workers or of individual connector? Which connector?
If it’s a sink connector, it’s just a consumer, so check if it’s committing offsets.
For source connectors, it’s possible there’s logic within the connector itself that checks versions into the offset store (the topic you defined in the workers), and it can decide to do whatever with that information.
If you change the offset config topic, then yes, all sources will get reset.
Thanks a lot, it is a source connector. The offset should have committed as I can see the offset content at the offset topic, also normal restart is able to retrieve the offset. However, if make changes to my code and make a new version of the plugin jar, the offset content is failed to retrieve when the connector starts up.
The connector is an in-house development, I don’t see any logic that checks the version into the offset store, not sure if there is some logic from the plugin architecture that prevents offset from getting retrieved if the jar file is updated?
I personally haven’t developed a source connector, but I cannot think of a reason that offsets would be modified on code changes.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.