Hi,
I have read about transaction API in Kafka. I want to know is connect API able to read database transactions. How to configure it?
Thank you
Kafka transactional API and database transactions are most probably different things (Confluent people, please correct me if I’m wrong here).
To have transactions in Kafka, this transactional logic must be implemented in your Producer, and then your Consumer should be configured to use read committed isolation level.
Theorerically, you can adjust Producer and Consumer configuration options for your Kafka Connect connectors, but this will not add the “transactional logic” to the underlying connector Producer.
I believe, that to make it possible, the connector itself should be able to capture “database transactions”, to replicate them to “Kafka transactions”. I personally don’t currently know of such connectors, which have this logic in their implementation.
Furthermore, even if you have this logic in your connector, this doesn’t mean that you will have “transactional data” on the Consumer side then. I mean, the Consumer will read only the committed data from Kafka, but this doesn’t mean, that “all or nothing” will be written to the target database for a particular transaction, unfortunately.
I think that this is something, which could be added to one of the future Kafka/Kafka Connect releases. To be able to transport data from one database to another in a “transactional” manner. However, I don’t know if this could violate the current principles, on which Kafka is built.
Thank you, This cleared most of the doubts.
1 Like
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.