We Replaced Kafka/Pulsar with a Woodpecker for Milvus—Here’s What Happened

This post details the replacement of Kafka and Pulsar with Woodpecker, a custom-built, cloud-native WAL system, in Milvus 2.6. It highlights the operational complexity, architectural constraints, and resource inefficiencies of using general-purpose message queues for Milvus's specific needs. Woodpecker's Zero-Disk architecture, which leverages object storage for logs and etcd for metadata, is presented as a solution for simplified operations, improved performance, and lower costs. Performance benchmarks demonstrate Woodpecker's superior throughput and lower latency compared to Kafka and Pulsar across local file system, MinIO, and S3 backends. The post also outlines Woodpecker's architecture components (Client, LogStore, Storage Backend, ETCD) and its two deployment modes: MemoryBuffer Mode for batch-heavy workloads and QuorumBuffer Mode for low-latency, high-durability deployments.