Apache Kafka Message Compression

This post details Apache Kafka's message compression capabilities. It explains how compression works in Kafka producers by batching data to the same partition, the impact of compression on throughput, network traffic, and disk utilization. It lists available compression algorithms (none, gzip, snappy, lz4, zstd) and provides recommendations (lz4 for performance, zstd for gzip-like ratio with less overhead). The post elaborates on configuring `compression.type` at both the topic and producer levels, detailing precedence and common/uncommon combinations. It also covers broker-side decompression scenarios and how consumers handle compressed messages. Finally, it offers advice on avoiding compression with encrypted data and inspecting compression codecs on brokers.