Perhaps I am using an anti-pattern here or I am missing something basic.
I am using streams to process a message which requires external API calls. If there is a failure that cannot be resolved (extended network outage), that message is placed onto a retry topic, where it is re-tried on a configurable interval (every 5 mins, for example). That all works.
My question is regarding handling a particular AVRO schema change. Specifically, when the incoming message adds a new REQUIRED field (FORWARD compatible topic). Of course, initially, everything will be fine. My application will continue to function, deserializing the new avro message into my old schema. That all works as expected.
The problem occurs when my application updates to the new schema and there are old messages in the retry topic. When my application originally placed that message on the retry topic, is placed it there with the OLD schema. So when I update and the application attempts to read it, a deserialization error will occur because my app now expects that new required field, but the old message in retry does not have it.
Does this make sense? This cannot be a new problem. I feel like I am missing something.
Thanks for any help.