Can I get the latest message for recovery?

Kafka Connect
1

HI my connector run in a distributed cluster, as single instance ( task.max=1 ) , say if the server running source task down, it is failover to the other server, and I would like the source task running at the new server pick up from where the first server down. There is a sequence number to indicate that, not sure how should I implement that?

Can I stamp the sequence number to the message published and I can get the latest message to get back the sequence number? Or I can just store the sequence number to kafka and retrieve it back? It look like there the offset is not related as there are multiple partition.

2

I am thinking probably the simplest way is read and write the sequence number in a separate kafka topic? Just wonder if there simpler way to get KafkaConsumer from SourceTask? Or I need to put the broker bootstrap.servers in the config again?

3

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.