Store changes from changelog topic using Kafka Connect

Kafka Connect Self-Managed Connectors
1

Can I persist data from changelog topic using Kafka Connect to a database?

2

Hi @SDev and welcome to the Confluent Community Forum!

This should be possible technically though I’m not sure how useful for your use case because the changelog topic record keys / values are binary and the serialization / deserialization is internal to Kafka Streams. So you’d really be writing byte arrays into a database.

Another potential wrinkle (depending on your use case) might be that the changelog topic is compacted. Would you need all records or would it be fine if the connector missed records that have been cleaned out?

What’s the use case for wanting to sink from a changelog topic into a database? Is it something like backup / restore where schema shouldn’t matter, or does schema matter? Which database?

Best,
Dave

3

Thanks Dave.

I have managed to persist changelog data successfully into Oracle DB. This is for long-term storage for regulatory and other reporting purposes