Send messages to sink connector only if value has changed

Kafka Connect
1

Hi,

I have a KSQLDB table that writes in a topic records with a simple boolean value.

I want to send them to another system, using a SInk connector associated to that topic.

The catch is, I only want to send messages if the value has changed, so:

VALUE
---------
true    -> Send record because last value is undefined
true    -> Don't send record because last value was the same
false   -> Send record because last value was different

So I was thinking to fix this using a custom transformation that would check if current value being sent is the same than the last value sent, but I’m concerned about concurrency issues when more brokers, connect instances or connector tasks are added.

Is there a simpler solution to this problem or the transformation is the only way?

Note: This is a follow-up question from the KSQLDB forum: KSQLDB table with a condition that involves several messages of a stream

Thanks in advance!

2

Kafka Connect isn’t designed to do this kind of stateful logic. You’re better off back in the #ksqldb (or #stream-processing ) area to implement this, with a nett output from the stream processor of only messages on a topic that you want Kafka Connect to push to the target system.

1 Like
3

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.