This is originally from a Slack thread. Copied here to make it available permanently.
You can join the community Slack here.
Vasco Ferraz
Hello there
I am using the SpoolDirCsvSourceConnector in order to ingest some CSV files.
In the connector I am defining the key.schema and value.schema.
Some fields must de defined has INT64 but some rows on the CSV are empty and I get the following error:
Could not parse '' to 'Long'which makes sense.I tried to add something like on the value.schema
"type": ["null", "INT64"]}but it seems that null is not a valid type and I get this error:
Cannot deserialize value of type org.apache.kafka.connect.data.Schema$Type from String \"null\": not one of the values accepted for Enum class: [STRING, INT16, STRUCT, BOOLEAN, ARRAY, FLOAT64, BYTES, MAP, INT64, INT32, INT8, FLOAT32]Any ideas how to solve this issue?
Thanks.
Neil Buesing
The “null” type is an avro construct that Connect API treats as optional
Connect’s Schema.java class
Schema OPTIONAL_INT64_SCHEMA = SchemaBuilder.int64().optional().build();I did a github search and found this example that might help
My GitHub Search if you want to look for others
extension:java SpoolDirCsvSourceConnector OPTIONALSo - something like (but properly escaped so it ends up as a string in your connect config (hence why I point you to what I found on GitHub)
"file_name": {"type": "INT64", "isOptional":true}
Vasco Ferraz
Thanks for the reply
Solution found. I need to add the following property into the connector:
"csv.null.field.indicator":"EMPTY_SEPARATORS"
Neil Buesing
well I am glad that connector has a solution to that; good find.
connectors are so much at the control of what they need to do that is unique to that connector that makes it hard to track down all the nuances — thanks for sharing the answer/fix.