We recently discovered that the HTTP Source Connector automatically converts all date fields to epoch format. This behavior can affect our data quality because some fields may contain values that look like dates but are not actual dates. We would like to know if there is a way to disable this feature. We are currently using:
name: kafka-connect-http-source
version: 0.2.6
The JSON parsing library that the connector uses doesn’t allow this behavior to be overridden AFAICT.
What’s an example incoming bad string and what does the connector do with it?
Possibly using TimestampConverter SMT to go back to string format would work.
thank you @dtroiano for your reply, With the TimestampConverter SMT, we can only apply it to fields of type date. Sometimes, an ID or code field may appear in a date-like format but is not actually a date. To address this, we are considering adding a parameter to our API to convert the payload to Base64, then using a custom SMT to convert it back to a string before producing it to the topic.
However, we are not certain if this is the best solution, as it would require adding custom code to our flow. I was wondering if there could be a parameter in the connector configuration, such as date.auto.epoch=true|false, that could handle this more natively — but I believe this is not currently possible, at least for now.