Get offset and partition values from topics into streams

ksqlDB
1

There is a topic which is based on Avro. I would like to stream it to another topic of JSON format. I am achieving it using two stream as follows

 CREATE STREAM AVRO_STREAM WITH (KAFKA_TOPIC='AVRO_TOPIC , VALUE_FORMAT='AVRO');

    CREATE STREAM JSON_STREAM WITH (REPLICAS = 1, PARTITIONS = 1, KAFKA_TOPIC = 'JSON_TOPIC', VALUE_FORMAT = 'JSON') AS SELECT *
    , AVRO_STREAM.ROWTIME "ROWTIME_COL"
    , TIMESTAMPTOSTRING(AVRO_STREAM.ROWTIME, 'yyyy-MM-dd HH:mm:ss') "ROWTM_TS"
    FROM AVRO_STREAM  AVRO_STREAM;

I want to get the OFFSET and PARTITIONS along with messages from AVRO_TOPIC into AVRO_STREAM and then into my JSON topic.

2

This isn’t currently possible. There’s an issue to track/upvote: Additional pseudo-columns for OFFSET and PARTITION · Issue #976 · confluentinc/ksql · GitHub

1 Like
3

This topic was automatically closed after 29 days. New replies are no longer allowed.