Hello guys, I’m trying to test Avro Union type within the same topic and TopicName Strategy
here is how I create topic and add reference for 2 independent schemas
confluent kafka topic create union-events --partitions 1
confluent schema-registry schema create --subject com.test.Pageview --schema ./schemas/pageview.asvc --type avro
confluent schema-registry schema create --subject com.test.Purchase --schema ./schemas/purchase.asvc --type avro
confluent schema-registry schema create --subject union-events-value --schema ./schemas/union.asvc --type avro --references ./schemas/union_refs.json
pageview.asvc
{
"fields": [
{
"default": null,
"name": "url",
"type": [
"null",
"string"
]
},
{
"default": null,
"name": "is_special",
"type": [
"null",
"boolean"
]
},
{
"default": null,
"name": "customer_id",
"type": [
"null",
"string"
]
}
],
"name": "Pageview",
"namespace": "com.test",
"type": "record"
}
purchase.asvc
{
"fields": [
{
"default": null,
"name": "item",
"type": [
"null",
"string"
]
},
{
"default": null,
"name": "amount",
"type": [
"null",
"double"
]
},
{
"default": null,
"name": "customer_id",
"type": [
"null",
"string"
]
}
],
"name": "Purchase",
"namespace": "com.test",
"type": "record"
}
union.asvc
[
"com.test.Purchase",
"com.test.Pageview"
]
union_refs.json
[
{
"name": "com.test.Purchase",
"subject": "com.test.Purchase",
"version": 1
},
{
"name": "com.test.Pageview",
"subject": "com.test.Pageview",
"version": 1
}
]
producer logic which I use to produce events
confluent kafka topic produce union-events \
--api-key "$KAFKA_API_KEY" \
--api-secret "$KAFKA_API_SECRET" \
--bootstrap "SASL_SSL://$KAFKA_BROKER_URL" \
--value-format avro \
--schema "some_schema_id" \
--schema-registry-endpoint "$SCHEMA_REGISTRY_URL" \
--schema-registry-api-key "$SCHEMA_API_KEY" \
--schema-registry-api-secret "$SCHEMA_API_SECRET" \
--verbose
```
when I’m trying to produce records it is not possible to serialize event like this
{"com.test.Pageview":{"url":{"string":"URL"},"is_special": {"boolean":false},"customer_id":{"string":"ID-003"}}}
I get an error like
Error: failed to serialize message: Union item 1 ought to be valid Avro type: unknown type name: “com.test.Purchase” or this Error: failed to serialize message: Union item 1 ought to be valid Avro type: unknown type name: “com.test.Pageview”
at the same time 'I’m able to produce the same events via Confluent Cloud UI and read them via confluent cli or Flink SQL
Could you help me to understand where the problem is ?
And my second question will be how implement the same but with TopicRecordName Strategy ? Do I need a union schema in this case or I just need to build topic-record.namespace.name subjects and on producer side choose correct subject name ?