Dealing with old schemas for intermediate topics

Kafka Streams
1

We’re moving our Streams app to work from a new topic with a new incompatible schema, but with the rest of the infrastructure in place.
When we do this, we have issues with it complaining of incompatible schemas in intermediate topics, though and we need to manually delete entries from the schema registry for all relevant intermediate topics.
My question is: when the app-reset tool is run, shouldn’t it also delete all entries from the schema registry for intermediate topics, or is there a setting for this already? Alternatively should we configure our Streams apps differently to avoid this issue somehow?

2

The application reset tool is shipped with vanilla Apache Kafka, and thus it does not integrate with Confluent Schema Registry. Seems like a nice addition we should ship with Confluent Community edition…

I don’t think there is much else you can do – one thing would be to disable the schema compatibility check for the topics in question in SR, but this would also imply that you lose the safe-guard SR provides, so I won’t recommend it.

3

Thanks,

We’ve just been deleting entries from the registry as we come across them which is a bit trial-and-error, but is enough for now whilst we don’t have too complex an ecosystem.

One other thing that would be useful is having something on this in the Confluent documentation - it came to us as a bit of a surprise.

1 Like
4

Did you try the “dry run” option of the reset tool to get the list of internal topics?

5

We updated the docs (Kafka Streams Application Reset Tool | Confluent Documentation)

The application reset tool does not:

[…]

  • Delete schemas in Schema Registry for internal topics. You must delete schemas for internal topics manually if you reset an app that uses Schema Registry. The reset tool has a “dry run” option you can use to see the internal topics that the tool will delete.