Avro Schema Evolution with a Retry Topic

[beattyk1]

Perhaps I am using an anti-pattern here or I am missing something basic.

I am using streams to process a message which requires external API calls. If there is a failure that cannot be resolved (extended network outage), that message is placed onto a retry topic, where it is re-tried on a configurable interval (every 5 mins, for example). That all works.

My question is regarding handling a particular AVRO schema change. Specifically, when the incoming message adds a new REQUIRED field (FORWARD compatible topic). Of course, initially, everything will be fine. My application will continue to function, deserializing the new avro message into my old schema. That all works as expected.

The problem occurs when my application updates to the new schema and there are old messages in the retry topic. When my application originally placed that message on the retry topic, is placed it there with the OLD schema. So when I update and the application attempts to read it, a deserialization error will occur because my app now expects that new required field, but the old message in retry does not have it.

Does this make sense? This cannot be a new problem. I feel like I am missing something.

Thanks for any help.

[beattyk1]

Well, I have settled on a two-step approach. First, add the new field to the schema as optional and provide the schema to both consumers and producer. Producer will always populate (because it will soon be required). Let it flow through the system. In this way all messages actually will have data in the optional field.

Once all old messages, without the optional field, are gone (our retention is only a few days), change the schema to required, then update the producer and consumer. Consumer will be fine temporarily reading messages produced against the “optional” schema as long as there is always data present in that field.

The key was that the deserialization will work when a schema says “required” but the data it is reading was produced as “optional”… as long as that optional value is there.