Reading files into Kafka using regex for filename

[riferrei]

Check this other discussion thread here:

FileStreamSourceConnector

I’ll echo what @riferrei stated. If you have files on a file system(block level access), use FileBeat. It’s going to be easier and more reliable in the long run. Object stores, like S3 or HDFS, are a bit different as they have different qualities and notification systems.

TL;DR: avoid using Kafka Connect for such relatively simple tasks like loading files into Kafka. Simple things should be kept simple: use Filebeat instead.

— @riferrei