Mode:timestamp+incrementing when incremented value is in wrong order

Kafka Connect
1

We have a table with which is maintained by two or more nodes. The ids are an incrementing number but since we have two or more nodes inserting the values of those ids might not always be inserted in the correct order. It could be like this.

UpdatedTime, id
10:00, 1
10:00, 2
10:01, 101
10:01, 102
10:03, 3

When we use the kafka connect jdbc source connector with mode:timestamp+incrementing on this table. Is that a problem?

The sql the kafka connector uses is this.

SELECT * FROM "public"."thetable" 
WHERE "public"."thetable"."updated" < ? 
  AND (("public"."thetable"."updated" = ? AND "public"."thetable"."id" > ?) 
             OR "public"."thetable"."updated" > ?) 
ORDER BY "public"."thetable"."updated","public"."thetable"."id" ASC
2

I don’t know for sure, but if you want to capture every event I’d definitely be looking at log-based CDC (Debezium, etc). Ref: No More Silos: Integrating Databases and Apache Kafka

3

Hi rmoff, thx for replying.
When we use the jdbc source connector we can determine how the kafka messages are built by creating views which it reads from. If we use the cdc connector we just get the raw tables sent to kafka with internal keys and so on, thats why we are still using the jdbc source connector. But it happened that we lost a few messages and thats why we try with the mode:timestamp+incrementing instead of just timestamp.

4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.