S3 Source Connector

Kafka Connect Managed Connectors
1

Hi all,

Need some advise/information

I have 3 outbound data streams (JSON docs) being written to a AWS S3 bucket.
(Don’t know yet if it will all go into one directory or 3 directories)…

but if into 1 bucket/directory… Can I use 1 source connector to ingest the 3 directories and then place the JSON contents onto 3 topics,
or
will it only go onto 1 topic.

other words can a source connector do some conditional filter/routing ?

if it all goes into 1 directory, can consume that one source and then route it onto 3 topics, using one configured source connector ?

G

2

S3 Buckets only have prefixes, rather than directories. Common mistake, but important qualifier.

Source connectors write to one topic, by default, but you can use SMT to conditionally modify the destination topic

3

heheh call it a prefix, but when ever any one sees a structure that’s /name/anothername/andantheone/filename.x they will see those names as a directory structure, used to “contain” a set of related files.

:wink:

G