Squeezing the firehose: getting the most from Kafka compression

What this post added

This post details Cloudflare's experience with Kafka compression, specifically focusing on the improvements introduced in Kafka 0.11.0. It highlights the challenges faced with older versions and the benefits of the new batch compression strategy. The post includes performance testing of different compression codecs (gzip, lz4, snappy) on various data types (nginx errors, capnp-encoded requests) and analyzes the trade-offs between CPU, disk, and network usage. It also touches upon the integration of these improvements into their Go-based Kafka clients (sarama) and identifies potential bottlenecks in the compression process.