My understanding is that when using the Schema Registry, consumers don’t need to be explicitly updated with a new schema (via a code/configuration change) because whenever they encounter an unknown schema ID in a message, they’ll pull that new schema from the registry.
Given that’s the case, why is it important to enforce old-reader compatibility (AKA forward compatibility)? It seems to me like there isn’t really such a thing as an old reader when using the Registry, since consumers automatically update themselves. Would you enforce old-reader compatibility just in case there are consumers that aren’t using the Schema Registry? Or is there something else I’m missing?
Keep in mind that compatibility checks happen on registering a new version of a schema, not when consuming. Compatibility checks provide assurance on the kinds of schema changes allowed, either incrementally (e.g., FORWARD) or compared to all prior versions (e.g., FORWARD_TRANSITIVE). Those compatibility guardrails imply the order in which you should update clients when a schema evolves.
In other words, the consumer code doesn’t directly care about compatibility (it doesn’t check and enforce). The consumer developer cares about compatibility because it tells them the kinds of schema changes that are allowed and the order in which to update producers / consumers when they do change the schema.
Thanks for your reply, dtroiano. I’m still not totally clear on this, though. You said, " The consumer developer cares about compatibility because it tells them the kinds of schema changes that are allowed and the order in which to update producers / consumers when they do change the schema." If my consumers are using the Schema Registry, I don’t have to do anything to update them, right? As soon as I update a producer the consumer will start receiving messages using a new schema and will automatically fetch that new schema from the registry. Why would I care that schema version 2.0 is incompatible with version 1.0 if my consumer will automatically update to 2.0 as soon as it receives a version 2.0 message?
As a developer, you’d care about compatibility because your application code might depend on it. For example, let’s say you aren’t using any compatibility checking (compatibility type NONE). Version 1 of your Customer schema has an int field called birth_year. Your application expects this field and will break without it – maybe it’ll hit an exception, maybe it’ll continue to run without hitting an exception but it’ll manifest as a logic bug. Now imagine someone on the producer dev team side decides that birth_year was a bad choice and they want a less limited birth_date field instead. They update the schema with this change (successfully since there are no compatibility checks). Then they update the producer in production. The consuming app doesn’t get updated, maybe didn’t even know about the change, so it breaks in one of the ways mentioned.
Now, instead, introduce compatibility checking into the scenario. E.g., maybe you specify FORWARD compatibility. Then the person who tries to delete the birth_year field won’t be able to and they’ll have to reconsider, or discuss the problem with partner dev teams to solve it. Or, maybe you specify BACKWARD compatibility. Then the birth_year field deletion is allowed by Schema Registry, but as an org you all know that you must upgrade your consumers first because you don’t have any assurance that consuming apps will work if you were to update producers first. The first example with FORWARD compatibility is a “hard” assurance in that the new schema will fail to register, and the second example is a “softer” assurance in that it guides how developers should coordinate upgrades but it’s still up to people to abide. You don’t get any of these assurances with compatibility type NONE (or if you imagine Schema Registry compatibility checking weren’t a feature).
To put what you’re saying in my own words, FORWARD compatibility check isn’t necessarily about making deserialization safe. It’s about protecting the data contract between producing and consuming applications.