Hello everyone!
I’m wondering if the S3 Source Connector is able to insert .csv files in a topic.
Another question is if with a consumer it is possible to read this topic and extrapolate the data of each file.
Thank you
Kafka isn’t really intended for file transfers, and CSV is not a recommended file format for structured serialization within Kafka Connect, as every field is a string.
Instead, perhaps would be better to ingest your files into RDS, for example, then use JDBC Source or Debezium to read from that.
Otherwise, you could use Spark/Flink to easily read CSV data from S3, then write to Kafka.
1 Like