How to pause and resume streams periodically

Kafka Streams
1

Hi I have a question. I have 2 topics. I want to periodically transfer a message from one topic to another.
For example every 15 minutes. I want to use kafka streams for this transformation.
When I stop streams, messages should not be streamed from topic to stream and I should be reopen message migration after 15 minute.
But I can’t achieve this because closed stream cannot be opened again. Is there a way sending message one topic to another periodically using kafka streams?

2

I am not sure if this is a very good way to use KafkaStreams. The framework is designed for stream processing, that means continuously updating data, not periodically updating data. Maybe you can consider using the Consumer and Producer APIs directly.

Anyhow, I responded to the concrete pausing / resuming question on SO: java - How can I pause (turn on/off) stream processing periodically(every 5 minutes) using Kafka Streams and Spring Kafka Streams? - Stack Overflow

1 Like
3

using the Spring Cloud Stream (if you are into Java), the framework provides StreamBridge component
you can use it to pause and re-start

1 Like
4

Instead of calling KafkaStreams#start() after .close(), you can just create a new KafkaStreams() instance and start it. On close() offset will be committed, so when you start the new instance (using the same application.id) it will resume where the first left off.

It might also be possible to use removeStreamThread() to scale the client down to zero background threads (instead of closing it), and resuming via addStreamThread.

5

Thanks. I tried removeStreamThread and addStreamThread method works :slight_smile:

1 Like
6

Thanks. I tried your answer and it works :slight_smile:

1 Like