Transform/FlatMap single event into multiple events

Kafka Streams
1

I have an incoming stream that I need to transform. I would like to map each individual event into multiple events. Each new event will have its own new key. I am using a mix and match approach using both PAPI & DSL APIs. In the example shown below, the message parts is a Java Enum.

Example:

Source event

MSG_ID MSG_SOURCE MSG_Date
1 ************ 06/3/2021
2 ************ 06/03/2021
3 ************ 06/03/2021

New Events:

NEW_KEY MSG_ID MSG_SOURCE MSG_Date MSG_PART
1|MP-11 1 ************ 06/3/2021 MP-11
1|MP-12 1 ************ 06/3/2021 MP-12
1|MP-12 1 ************ 06/3/2021 MP-13

2|MP-21 2 ************ 06/3/2021 MP-21
2|MP-22 2 ************ 06/3/2021 MP-22

3|MP-31 1 ************ 06/3/2021 MP-31

2

What is your question? What did you try (code snippet)?

I am using a mix and match approach using both PAPI & DSL APIs

What is the exact problem?

For example, there is KStream#flatMap() that should help.

3

The goal is to produce multiple events from the source event. I need to make sure that events are not produced twice. That is the reason I need to use a State Store. I am not sure if I can return multiple records as the output of a transform call. Can I return a list of new events for each input event?

4

I am not sure if I can return multiple records as the output of a transform call.

For KStream#transform() you can could call context.forward() multiple times. However, it’s not type safe. Instead, you can use KStream#flatTransform to return a list or records (similar to flatMap()).

1 Like
5

flatTransform will work for my case. Thanks

1 Like