Streaming vs micro-batching

This could be a common question, so copying this over from this post in Slack.

Pavel Rephiah
I will be happy if someone could point me to good reading about how streaming is different from micro-batching, meaning in technical point of view, it just making batches very small and more frequent kind of creating streaming by itself.

Gerard Klijs
To me it’s the same thing, as Kafka is already typically micro-batching.
:+1:

Alexei Zenin
Kafka consumers under the hood are polling so they are not streaming in reality as Gerard mentions (in terms of the events being pushed to you with no way for back pressure). Reading this Javadoc helps cover a lot of the concepts: KafkaConsumer (kafka 1.0.1 API)

Gerard Klijs
Indeed and the typical (high performing) producer will also bundle multiple messages in one tcp packet when the load is high enough.

Mitch Henderson
If you’re familiar with spark streaming it operates the same way as Kafka consumers.

A relevant (and really good) podcast from confluent that discusses (mentiones) microbatch vs streaming :

2 Likes