How Walmart Uses Apache Kafka for Real-Time Replenishment at Scale | Confluent How Walmart Uses Apache Kafka for Real-Time Omnichannel Replenishment

What this post added

This post details Walmart's implementation of Apache Kafka for real-time omnichannel replenishment. It describes their micro-batch architecture, which processes tens of billions of messages daily for nearly 100 million SKUs. Key technical aspects include using Kafka for ingesting data via Change Data Capture (CDC), denormalizing it for accessibility, and processing it in a planning engine. The system publishes plans through various Kafka topics for different consumers. The post highlights the scale of their Kafka deployment (18 brokers, 20+ topics, 500+ partitions per topic). Architectural considerations include real-time input streaming, multi-tenancy, enabling diverse downstream consumers, and ensuring tight contracts between inputs and the replenishment engine. Platform decisions like active-passive data replication, automatic VM recovery, Kafka retries, and an in-house audit/replay mechanism are discussed. Significant tuning and optimizations are detailed for both producers (custom partitioning strategy using murmur hash, linger.ms, batch size, acks='all') and consumers (max.poll.records, max.poll.interval.ms, disabling enable.auto.commit, session.timeout.ms, heartbeat.interval.ms).