Delete and Recreate Kafka Source Connector Consequences

sshankar.sateesh · 17 January 2022 09:12

Hello Team,

I have a question. I have a kafka source connector process which runs in prod and polls couchbase every 100ms . Looks like the seed node names are changing. I have to now delete the connector and update the connector with the new node names and recreate the connector. Question is if I delete and recreate the source connector with the same name, will it begin polling from the saved offset or will it start from now on causing data loss? Using Couchbase Kafka Connect 4.1

nbuesing · 10 February 2022 17:35

github.com

couchbase/kafka-connect-couchbase/blob/ac3d020ef647e778c3240e8c89c6808ad9faa757/src/main/java/com/couchbase/connect/kafka/CouchbaseSourceTask.java#L261


      
                sourcePartitions(partitions));
          
          
  LOGGER.debug("Raw source offsets: {}", offsets);
          
          
  for (Map.Entry<Map<String, Object>, Map<String, Object>> entry : offsets.entrySet()) {
              Map<String, Object> partitionIdentifier = entry.getKey();
              Map<String, Object> offset = entry.getValue();
              if (offset == null) {
                continue;
              }
              int partition = Integer.parseInt((String) partitionIdentifier.get("partition"));
              long seqno = (Long) offset.get("bySeqno");
              Long vbuuid = (Long) offset.get("vbuuid"); // might be absent if upgrading from older version
              partitionToSequenceNumber.put(partition, new SeqnoAndVbucketUuid(seqno, vbuuid));
            }
          
          
  LOGGER.debug("Partition to saved seqno: {}", partitionToSequenceNumber);
          
          
  return partitionToSequenceNumber;
          }

looks like offsets are stored by seqno and vbuuid – if either or both of these are what couchbase calls “seed node” then yes, it looks like they would change.

I am not a couchbase user, but knowing how offset management is handled for Source connectors in Kafka Connect - this looks to be the code of interest for you to check out.

system · 12 March 2022 17:36

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Can a source connector update its offset without sending a record? Kafka Connect	1	2897	13 October 2022
Can a source connector update its offset without producing records? Kafka Connect	4	1547	12 November 2023
Connector offset reset if connector version change Kafka Connect	4	3367	17 August 2022
Do multiple source connector tasks create multiple distinct connect offsets? Kafka Connect	3	3363	21 August 2021
Easy way to reset Kafka Connect Connector Offsets Kafka Connect	4	4890	15 March 2021

Delete and Recreate Kafka Source Connector Consequences

Related topics