I am new to Kafka. I am using Spring Cloud Stream Kafka in my project, also I am writing code in functional way.
I want to consume a message from single input topic, build different models from message and publish each model to a different topic (i.e. 1 model → 1 topic) without using any predicates or branching using DSL.
Can someone please let me know if it is possible to achieve this in Kafka.
Update: Actually I found a way to do this. We can create multiple functions and each function consumes same message from same input topic but while producing we only publish particular model to particular output topic. I see below disadvantages in this approach
- I need to create multiple streams to publish models to output topics
- My business logic is same in all the functions, it generates all models every time and I need to select particular model that I want to sent to output topic.
Code
@Bean
public Function<KStream<String, TestEvent1>, KStream<String, TestEvent1>> testFunction1() {
return input->input; // publishes message to output topic
}
@Bean
public Function<KStream<String, TestEvent1>, KStream<String, TestEvent1>> testFunction2() {
return input->input; // publishes message to output topic
}
Configuration
spring:
cloud.stream:
function:
definition: testFunction1, testFunction2
bindings:
testFunction1-in-0:
destination: test.inputtopic
testFunction1-out-0:
destination: test.outputtopic1
testFunction2-in-0:
destination: test.inputtopic
testFunction2-out-0:
destination: test.outputtopic2