Is Kafka-connect appropiate for large volume migrations?

Kafka Connect
1

I need to migrate from one database to another where the existing volume of data is at least 2TB and at least 200 million messages. I also need to do some transformations as part of the migration as the destination database expects a different format

Both databases offer official connectors that I know that I can use

Is Kafka-connect recommended for this type of workflow?

The main challenge that I have is that I need to be able to migrate data of a production system that it is still receiving traffic

2

@OneCricketeer Any chance you can help me with this one?

3

I used S3 sink to store and archive hundreds of GB per day.

It worked fine for our use case, I can’t answer if it’ll work for everyone

1 Like
4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.