Unable to upgrade stream with Create or Replace

ksqlDB
1

We have a ksql stream originally created with an inferred AVRO schema value using Schema Registry:

CREATE STREAM STREAM1 WITH (
KAFKA_TOPIC = ‘db.stream1’,
KEY_FORMAT = ‘JSON’,
VALUE_FORMAT = ‘AVRO’,
VALUE_SCHEMA_ID = 100001
);

After evolving the schema 100001 by simply adding a new optional field, we are not able to use CREATE OR REPLACE to upgrade STREAM1 to use the new schema 100002.

The error returned is:

Cannot upgrade data source: DataSource ‘STREAM1’ has topic = KsqlTopic{…} which is not upgradeable to KsqlTopic{…}

Are schema upgrades supported by KsqlDB stream/tables? STREAM1 has many dependencies, so to drop and create again would require manually cascading most of our streams. Also taking this approach, are there any considerations we’d need to consider to prevent message loss?

2

Adding a new column is actually possible: How to update a running persistent query - ksqlDB Documentation

However, it requires you specify the full schema (cf ksqlDB 0.14.0 - Newest Features and Updates) – schema inference plus upgrade is not supported.

3

This topic was automatically closed after 30 days. New replies are no longer allowed.