Hi there! Say I have an Order table with a field order_status, which holds a foreign key to the OrderStatus lookup table, which contains the types of order statuses. When an Order’s status changes I would like Debezium to produce the change but replace the foreign key of order_status with the value from the OrderStatus lookup table. So instead of consumer’s just seeing an id for order_status, they see the string value instead.
After some research I’ve not found a definitive answer to solve this. Is this possible using Debezium configuration, or any other method?
EDIT: I would do this in stream processing, but as OrderStatuses is a static lookup table, it will not produce changes, so it don’t produce a stream. I also don’t want my streams processor to know how to map the foreign key to a string itself.
Debezium, like other log-based CDC tools, relies on entries in the transaction log of the source database. This means that it can’t do lookups on the RDBMS side like you are describing.
You’ve already identified that you could do this in stream processing. Just because OrderStatuses doesn’t change doesn’t mean that it can’t be a stream of events. This is why Kafka Streams and ksqlDB have the concept of tables as well as streams. You can then perform a stream-table join.
Thank you! I’m working through the tutorial. So if I setup Debezium to capture tables Order and OrderStatus, the initial snapshot of these tables will be produced to the broker. If I then start up my stream processor then it’ll consume OrderStatus snapshot and store it in the KTable? So this is a ‘one shot’ capture of the lookup table?
the initial snapshot of these tables will be produced to the broker. If I then start up my stream processor then it’ll consume OrderStatus snapshot and store it in the KTable? So this is a ‘one shot’ capture of the lookup table?
