Avro kafka to json stream

ksqlDB
1

i have a avro kafka. can i make a json stream from it immediately or i need to create a avro stream first and then the second stream as json as SELECT * from stream1?

I have an avro kafka, i have created a stream but the select doesn’t return anything from it.

i am wondering whether this structure is correct i put in the CREATE Stream is correct or not.

CREATE STREAM TcProductVariant_stream_AVRO (
_comment string ,
data ARRAY<STRUCT<
productInstanceUuid string ,
productInstanceId string,
productInstanceDescription string ,
productInstanceRevision string ,
productVariants ARRAY<STRUCT<operation string , TC_variant_id string , tc_variant_name string, variant_options ARRAY < MAP< string,string > >>>

)
WITH (KAFKA_TOPIC = ‘TcProductVariants’,
VALUE_FORMAT = ‘avro’,
TIMESTAMP_FORMAT = ‘yyyy-MM-dd HH:mm:ss’,
PARTITIONS = 6);

below is an example of message in the topic

{
“_comment”: “W\u0013\T6\u0014[/5Cw”,
“data”: {
“productInstanceUuid”: “b3a50b58-ec9b-46f8-aee5-50d9f9e6aa2d”,
“productInstanceId”: “P444555666”,
“productInstanceDescription”: “BMX Freestyle Pro”,
“productInstanceRevision”: “B”,
“productVariants”: [
{
“operation”: “Add”,
“Tc_variant_id”: “f8f3d986-b440-431e-9013-3bfdf6750bb2”,
“Tc_variant_name”: “red AND large”,
“VariantOptions”: [
{
“color”: “green”,
“size”: “large”
}
]
},
{
]
}
}

2

You’d need to do the latter with a second stream. Here’s a tutorial converting Avro to Protobuf that you can follow, where you’d target JSON or JSON_SR rather than Protobuf.

3

thank you for your quick reply.

that was the exact link i saw that made me to submit the question.

In there, it has a stream on the topic and then on the second stream it changes to protobuf with select * from stream1

i did build two streams from my avro tppic: json and avro.
what is the criteria for me to choose which format is appropriate for the stream?

also, my message is very nested. I need to flatten it to make it easier for sinking . I am planning to create other set of streams or create tables after these streams. I don’t know criteria to choose stream or table. is there anting you can suggest criteria to pick table versus stream for further processing? .

4

This topic was automatically closed after 30 days. New replies are no longer allowed.