Schema association shows null value

[DaniWS]

Hello all,

We’re trying to delete unused schemas to reduce costs using the schema deletion tool as per the docs. However, we noticed that our topics show a schema associated with them, but when we go to those schemas, they show “Associated with topic: --”, as if they’re not associated to any topics.

Hence, we’re hesitant to delete these “unused” schemas are they seem to be associated to topics. This seems like a contradiction.

Anyone knows why this happens, and how can one make sure that the schema is really not in use by any topic in order to delete it?

See screenshot below for an example.

Thanks

[dtroiano]

I’m asking around to get more info about how we populate the Associated with topics list.

In the meantime, if you consume that topic and print schema IDs, will you actually see the schema being used? Follow the Confluent Cloud steps here to print schema IDs (Note: the usage of SCHEMA_REGISTRY_URL as the bootstrap servers endpoint in the kafka-avro-console-consumer command and credentials config file is incorrect. We’ll fix that. You’ll want to use the bootstrap servers endpoint instead).

[dtroiano]

@DaniWS if you refresh the page do you still see Associated with topics: --? I am seeing that if I create a schema t-value, then topic t, there’s a brief period of time (a few seconds) where the UI shows Associated with topics: -- but then if I refresh the page the association shows up. Is this the case for you or does it last longer? If it’s longer, can you share your order of operations wrt topic / schema creation and producing messages?

The Associated with topic list itself is only based on TopicNameStrategy whereas the schema deletion tool is message-based in determining which schemas are in use. It’ll actually scan all topics from their beginning offsets and scan event keys/values for schema IDs. So, you wouldn’t necessarily expect the tool to agree with what you see in the UI. This is pretty confusing so you might expect the UI to clarify topic association in the future. This is how it is at the moment, though.

[DaniWS]

Hi, Dave.

Thanks for your response. To expand a bit, the topics of this cluster have existed for a long time, and the producers/config hasn’t change in some time as well. The schemas I see on each topic information in the console match the naming convention, i.e. topic: some-topic-name schemas: some-topic-name-key some-topic-name-value.

I also get the schema ID of some-topic-name-value under the topic information (for most topics I’ve checked). However, these schemas show “Associated with: --” in the Console. The schema deletion CLI tool also list them as unused and candidates for deletion.

I will try to consume from one of these topics and print the schema ID as you suggested. Will report back the results.

Thanks again.

[DaniWS]

@dtroiano I consumed some of the latest messages of a topic as indicated, printing the shcema IDs. It seems to me that the schema deletion tool is showing inaccurate results.

For example, when I consume from topic “production.dpt.rrss.client-matcher” using the kafka-avro-console-consumer tool, I get the schema ID 100007, which corresponds to version 2 of schema “production.dpt.rrss.client-matcher-value”.

If I go to this topic information in the Confluent Cloud console, I also get this same schema ID: 100007 (even though it shows “Associated with topics: --”).

However, the schema deletion tool shows this schema as not used by the topic, and candidate for deletion, i.e.:

Scanning topic production.dpt.rrss.client-matcher from cluster XXX-XXXXX...
Following 1 schemas are unused schemas that qualify for deletion.
+---+----------+----------------------------------------------------+---------+
|   | SCHEMAID | SUBJECT                                            | VERSION |
+---+----------+----------------------------------------------------+---------+
| 0 |          | production.dpt.rrss.client-matcher-value | 2       |
+---+----------+----------------------------------------------------+---------+
Please select the schemas you want to delete by typing the numbers (1st column), separated by comma:

Am I missing something here? Or the schema deletion tool is not working as expected?

Thanks!

For reference, the command used to consume from the topic was:

➜ ./kafka-avro-console-consumer --bootstrap-server XXXXXXX
–property print.key=true --property print.schema.ids=true
–property key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
–consumer.config ./client.properties
–topic production.dpt.rrss.client-matcher --partition 0 --offset 3161797 --max-messages 15
–property basic.auth.credentials.source=“USER_INFO”
–property schema.registry.url=https://XXXXXXXX.eu-central-1.aws.confluent.cloud
–property schema.registry.basic.auth.user.info=XXXXXXXXX:XXXXXXXXXXXXXXX

[dtroiano]

@DaniWS I’m able to repro similarly confusing behavior. I’m investigating to see if there’s a misunderstanding or if there’s a bug in the schema deletion tool.

[dtroiano]

@DaniWS this looks to be an incompatibility bug between the plugin and version 4 of the Confluent CLI. I opened a PR here where you can follow the fix.

[dtroiano]

@DaniWS the fix has been merged, so you can pull that down and rebuild / reinstall the plugin. Thanks again for reporting this!

I am still not sure about the Associated with topics: -- issue for a long-existing topic. I’m asking around about that.

[DaniWS]

Thanks Dave,

As long as the CLI plugin tool works as expected that is fine for me. So I’ll update and give it a try and report if it worked.

Thanks a lot!

[DaniWS]

The schema I was testing on, which was being reported as unused by the tool, while it was indeed being used, is no longer showing as used.

So the update seems to have fixed the issue I assume.

Thanks a million! Could have never guessed the cause of the problem myself.