Sending messages based on preferences

I was looking for a way consult DB data before acting on messages in a topic. Administrators change these preferences and we want to check them before we send these messages. This is custom logic to send emails etc.

I was thinking the data from the Oracle DB can be refreshed using CDC. So only the new preferences with the latest timestamp will be consulted from the topic. Older ones are outdated. Is this the right way ? I don’t want to issue DB queries in the Consumers.

Can I load the preferences in a KTable based on the latest timestamps and consult it ? I can drop messages that don’t conform to the preference rule.
Moreover calling a DB or any external service in the streams library was discouraged.

What should I do with the older preferences with old timestamps ? Do they now clog the topic ?

These preferences are just data from the DB and they may change but they are not considered real-time streaming data. Admins. change it periodically.


Yes, you can have a topic for the preferences, and have it ‘compacted’ so only the lastupdate is kept.

Then either use a Kafka Streams join, via a Ktable to join the messages and the preferences.