What if I have a field that I want to both cast to a different type AND preserve the original value?
I have an Avro key which includes some nullable fields. I want to pipe these messages into Postgres using the JDBC Sink Connector using the key as primary key. I want to do that so that I can use “delete.enabled” configuration for the Sink Connector. All of that works but obliges me to make the nullable fields not-nullable. (Can’t include nullable columns in a primary key)
So one idea that came to mind was:
- keep the fields nullable strings in Avro
- cast the fields to non-nullable strings in Sink connector config (possible?)
But even if that were possible (I haven’t found a coalesce option for SMT so far), that’d have the undesired side effect of obscuring which messages had non-null pre-cast values vs which messages happened to have values matching the cast’s output.
Is there a way to copy the nullable string to another field in the message value, while also casting it to a not-nullable field within the key?
(I know I could achieve this via ksqlDB or Kafka Streams, piping the data into a secondary topic. I’m just curious if anyone knows of a way to achieve this via off-the-shelf SMT in Kafka Connect)