Have delete in source system reflected as a soft delete in destination

Kafka Connect
1

Use case: A row is deleted from the source system and want to reflect the deletion as a soft delete in destination. Basically, I want to flip a flag in the destination system. Is this something that is possible with Kafka Connect?

2

What specific systems / connectors are you using? I can think of some examples where this would be doable, e.g.:

  1. Debezium CDC source connectors give you delete events (e.g., see here for Postgres)
  2. The JDBC sink connector supports upsert insert mode where you can flip a flag of an existing DB record

Depending on the source / sink records and connectors used, you might have some record transformation to do in between, and you can either look at SMTs (single message transform) on the connector(s) or a more fully featured stream processing solution in the form of a Kafka Streams or Flink application.

3

These are moving data between 2 relational databases. I’m using the jdbc connector.

4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.