Hi @rmoff and thank you for your response. We are in the evaluating Kafka for many of our processes.
Yes, I want to ingest a CSV from a customer into Kafka and then a db much like you do in hour video. Loading the data is just the first step in our process. Upon receiving customer data we then send a subset of this data in various forms to vendors who then send the data back - also in different forms.
Once we receive everything from our vendors we merge the data and then send back to our customer - and yes, in different forms based on what they’ve requested (i.e. JSON or CSV).
I’m currently investigating how to trigger consumer processes to send the data to the vendors. Using your solution, I may rename “processed” to something like “written to db” and then possibly have another connector which monitors this folder to grab the filename and last modified date and write to a topic that my “vendor” consumers suscribe to. Does this make sense?
Also, when ingesting the data into kafka and then the db - is there a way to add the name of the file and the last modified date so that this data is written to each row? The consumers would then pick up the message with the filename and date and query the db with this information to create the vendor outputs.
Thanks so much for your work and guidance. I just started with Kafka yesterday and after troubleshooting “not an absolute path” errors due to not having created the volume corresponding to the $PWD, I’m up and running.
I’m currently using the spooldir plugin - I will look into the second connector (file-pulse) and how I might be able to add the filename and the file’s last modified date to the payload for each row that will be written to the db. Surelly someone has done this before