How to remove deduplication upon struct in ksqldb

ksqlDB
1

Hi ,
Step 1 i am creating stream with create Type with struct format as json

CREATE TYPE contact AS STRUCT<"otherPhone" VARCHAR, "workPhone" VARCHAR, "address1" VARCHAR, "city" VARCHAR, "country" VARCHAR, "zipCode" VARCHAR>;

CREATE STREAM event_load(KEY VARCHAR KEY,"eventType" VARCHAR, "platform" VARCHAR, "billingAccount" BILLINGACCOUNT, "soldToContact" CONTACT, "billToContact" CONTACT, "subscription" "subscription") WITH (KAFKA_TOPIC = 'topic1.in', VALUE_FORMAT = 'JSON',PARTITIONS=1);
upon this i am applying  select * 

CREATE STREAM dsc_events_supply WITH (KAFKA_TOPIC = 'subscription.in, VALUE_FORMAT = 'JSON',PARTITIONS=1) AS select * from event_load where "eventType" in ('SubscriptionCreated') ;

so here is the problem: when I alter the stream event_load with add column and dropping stream dsc_events_supply and recreating it for addition of new column from event_load in my subscription.in topic data is loaded twice: 1st set of data which was already existing and second set of data with new added column, so duplication is happening. I am not able use this How to find distinct values in a stream of events using ksqlDB because my data is struct making group by columns into primary key. So how to fix this issue

2

This topic was automatically closed after 30 days. New replies are no longer allowed.