Apache Kafka for Real-Time Retail at Walmart Labs | Confluent

This post details Walmart's adoption of Apache Kafka for their 'Item Setup' process, which involves managing products, offers, pricing, inventory, and logistics. It describes how Kafka was used as a central data hub to connect hundreds of microservices and teams, modeling changes as event streams. The implementation leverages stream processing technologies like Apache Storm and Spark, plain Java programs, reactive microservices, and Akka Streams. The post highlights the scale of operations, processing billions of updates daily, and the challenges encountered, such as managing service topologies, schema management, multi-DC data balancing, and shielding consumers from business-impacting changes. It also mentions the development of operational tools for tracking flows, SLA metrics, and latency, and the potential for reprocessing catalog data, sharing data between environments, A/B testing, and analytics.