File Pulse Source Connector (Multiple topics per connector configuration)

Kafka Connect
1

I Need some guidance/information on FilePulseConnector.
I have to read multiple files belonging to 30 different file formats.
I want to configure separate file input reader for each file format and want separate output topic for each file format.
As per my understanding, for the given list of files which are filtered, the File pulse source connector supports common configuration for File Input reader and output topic.
And If I want to deploy this is distributed mode, does that mean I have to configure 30 different Kafka Connect clusters for same? - Please correct if i am wrong…

2

Hi @utkarshnehete,

The FilePulse source connector supports dynamically overriding the output topic using the built-in AppendFilter, for example:

...
  "filters": "TopicRoutingFilter",
  "filters.TopicRoutingFilter.type": "io.streamthoughts.kafka.connect.filepulse.filter.AppendFilter",
  "filters.TopicRoutingFilter.field": "$topic",
  "filters.TopicRoutingFilter.value": "foo",
...

If you can pull the desired output topic from the record or other available metadata then this should work for the topic routing part of your question.

Separate file input reader is not doable AFAICT, though. So to do that, at a minimum you’d need a connector per reader.

And If I want to deploy this is distributed mode, does that mean I have to configure 30 different Kafka Connect clusters for same?

You would need multiple connectors that can run in the same Connect cluster, not multiple Connect clusters.

HTH,
Dave

3

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.