Can a source connector update its offset without sending a record?

Kafka Connect
1

Hi!

I maintain the Couchbase Source connector, which can be configured to ignore certain upstream events. Even if an event is ignored, the connector would like to tell Kafka Connect it has a new source offset, even though there’s no message to send.

Is there an API or recommended technique to update the source offset without publishing a record?

The best workaround I can think of is to publish a dummy message that gets dropped by a Single Message Transform. (This would still result Kafka Connect updating the source offset, right?) Is there a better way to do this?

Thanks,
David Nault

UPDATE: After a bit of experimentation, it looks like the workaround of using an SMT to drop the dummy record doesn’t work, because Kafka Connect doesn’t save offsets of records filtered by SMTs. Is that correct?

[2022-09-13 19:22:54,717] INFO [test-couchbase-source|task-1|offsets] WorkerSourceTask{id=test-couchbase-source-1} Either no records were produced by the task since the last offset commit, or every record has been filtered out by a transformation or dropped due to transformation or conversion errors. (org.apache.kafka.connect.runtime.WorkerSourceTask:502)
2

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.