Months ago, we have our Kafka service host on cloud and manage by a “company”
One day, we got alerts to let us know the Kafka nodes were down as well as Schema Registry Server.
Once the Kafka nodes back on service, our stream apps could not process un-processed data because we lost the many schemas.
Even though we control the schema but we DO NOT have control on the schemaId when we create that schema on the server.
As the result, we tried many options but could not do anything with those unprocessed data since we could not create the new schema, and guaranty will have the same schemaId as before.
Finally, we delete all unprocessed data and start over.
Now the question is what option do we have if we do not have access to the old schema-registry?
In the past, I have thought about this by switching to use JSON schema instead of Avro.
However, process and store billion records with JSON vs Avro leaves no choice to pick AVRO to save disk space.
Has anyone from Confluent thought about overwriting the schemaId when creating a new schema?
To me, we should allow us to create schema and be able to override the schemaId if we want to.