Streams evaluation

Kafka Streams
1

Hello,
We have different types of messages(SMS,Emails etc.) being sent to consumers from the Kafka Consumer. When I read about the Streams API the use cases are usually related to stream processing. We want to use a scalable API but at this time we are not operating(map/reduce/filter) on the messages and creating topics in the consumer.
But we need DLQs and we can’t anticipate the future changes that may require Streams. And we need to monitor the messages carefully as it is a financial system. We don’t want to lose messages without trace or send duplicates.

Should I look at the Streams API(Spring Boot) more closely for our use cases ? What specific technical features apply to our type of systems ?
Events are coming into our system through Kafka Connect.

Thanks

2

You probably want a plain consumer rather than KStreams which reads those messages and then does the action to then forward to other topics or dlq. Usually we don’t do retries and things like that in kstreams and especially not doing external calls like email or SMS which might fail and need a certain number of retries before giving up.

3

Can I not put the failed messages back into the Stream with a count in the message itself ?
If it exceeds it can be dropped.
Why are they unsuitable for external calls ? The business logic is specific to the application.
Thanks.