Lets say I use the SpoolDir Source connector for disgesting a CSV files. This SCV file has a field named EventTime. This field is formatted as yyyy-MM-dd HH:mm:ss.
Is it possible to use either a SMT or something else to break this field up in 2 (e.g. Date & Time) and populate these fields using Date part and Time part of the EventTime field?
Raoul
Can you give some context as to why you’d want to do this?
If you have a perfectly good timestamp it doesn’t strike me as the best idea to break it apart at ingest; instead use a function when processing to extract the date / time as required.
Robin,
We are planning to use Kafka in combination with Spark and there was a request to have the date time split. I thought it might be easiest to do this as close to the source as possible. But instead I believe we will do this on the spark side when we consume the data from Kafka.
Was just pushing out a random thought.
Thank you.
Raoul.
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.