Apache Kafka for Real-Time Retail at Walmart Labs | Confluent

What this post added

This post details Walmart Labs' adoption of Apache Kafka for their 'Item Setup' process, which involves normalizing, matching, classifying, and processing product data, offers, pricing, inventory, and logistics. They model changes as event streams recorded in Kafka, processed by technologies like Apache Storm, Spark, plain Java, Reactive Microservices, and Akka Streams. Kafka serves as a company-wide data hub, enabling hundreds of microservices and teams to operate autonomously. The post highlights the growth in Kafka usage, with billions of updates processed daily, and the development of operational tools for monitoring SLAs, latencies, and backlogs. Challenges include managing service topologies, schema management, multi-DC data balancing, and shielding consumers from business-impacting changes. The adoption has enabled agile development and faster onboarding of sellers and product listings, with Kafka also powering a near real-time search index.