I have a given scenario where the kafka source connector checks if the file is duplicate based on the hash of the file, for example if I have two files tc1 and tc2 they have same records in them it will thus generate a same hash. If such is the case then I would log a warning stating Duplicate file received.
I wanted to use offsetStorage interface and use it like below:-
partition = ImmutableMap.of("test", recordChecksum()); Map<String, Object> offset = this.context.offsetStorageReader().offset(partition);
Where recordChecksum() returns the String which is the hash value of the file received by source connector. But for some reason each time I check offset value it is always null even though my sourcePartition is populated. It seems like I need to activate the offsetStorage for Connectors? As no matter what I do it never gets populated. Please help ?