Message sizes in Kafka

Kafka was not built for large messages. Nevertheless, more and more projects send and process 1Mb, 10Mb, and even much bigger files and other large payloads via Kafka. One reason is that Kafka was designed for large volume/throughput – which is required for large messages. It works well for image processing, video streaming, and sending (or even processing) large CSV or XML files.

Here is a blog post with many details (config, architecture, use cases) about handling large messages in Kafka:

Handling Large Messages with Apache Kafka (CSV, XML, Image, Video, Audio, Files)

1 Like