SMT TimestampConverter not maintaining precision

Kafka Connect Managed Connectors
1

Hi Everyone,

I’m doing a Timestamp transformation in my data pipeline however i lose microseconds precision here is my SMT code block

“transforms”: “TimestampConverter”,
“transforms.TimestampConverter.type”: “org.apache.kafka.connect.transforms.TimestampConverter$Value”,
“transforms.TimestampConverter.format”: “yyyy-MM-dd HH:mm:ss.SSSSSS”, “transforms.TimestampConverter.target.type”: “Timestamp”,
“transforms.TimestampConverter.field”: “creation_date”

creation_date in kafka topic looks like this 2023-07-26 12:34:56.123809 but on my postgres database it loses precision. Here is how the creation_date looks like in my database 1970-07-26 12:36:59.809000 after being transformed. can you please assist

Thanks

2

Unix epoch time, which is documented as being used by this converter, is measured in seconds, not microseconds

1 Like
3

Thanks for your reply @OneCricketeer, meaning the is no way i can get the micro-seconds using this converter?

4

You’d need to write/use a different transform first that divides by 1000, and drops precision.

You can see the documentation explicitly says Unix epoch (which are seconds, as mentioned)

5

Alternatively, keep the data as a number. Then, it’ll occupy less storage on the broker, and over the wire. Then do whatever conversions you need within your consumers.

1 Like
6

Thanks @OneCricketeer, i will have to write a custom converter