Handling Large Messages with Apache Kafka

(Originally posted on Kai Waehner’s blog: Handling Large Messages with Apache Kafka (Image, Video, Audio, Files)

Kafka was not built for large messages. Period. Nevertheless, more and more projects send and process 1Mb, 10Mb, and even much bigger files and other large payloads via Kafka. One reason is that Kafka was designed for large volume/throughput — which is required for large messages. This blog post covers the use cases, architectures, and trade-offs for handling large messages with Kafka.