I got not yet consumed message in the snowflake sink connector. This issue only happens when i delete and recreate the topic. Initially when i create the topic for the first time it works fine as expected. But then when the topic is recreated I am seeing this message in the snowflake sink connector. This prevents from writing into the target snowflake table. Topic has the data and also i am able to read it.
Below is the full snowflake sink connector
tablename-0/OffsetAndMetadata{offset=3968616, leaderEpoch=null, metadata=‘’} – not yet consumed, taskOffset=3968616 currentOffset=451339 (org.apache.kafka.connect.runtime.WorkerSinkTask:430
Additionally when I describe the consumer group the current offsets are all set to - which seems to indicate that it never starts consuming anything at all. I’m using version 2.2.0 of the connector.
Update: For me this seemed to be because I attempted to resync the data back to a table using snowpipe streaming but I did not drop the table and let it get recreated prior to attempting the sync. I did remove all of the data in the table but the table itself was still there. The documentation Snowpipe Streaming | Snowflake Documentation seems to suggest that some offset information is stored inside of snowflake table and maybe that’s why it would no longer sync.
Thank You Alex for this information. In my case topic was removed and created again. Even i was able to load data into the newly created topic with the same name as before. It seems like it is more on the snowflake side for this one.