Owl: Distributing content at Meta scale
Jul 14, 2022Jason Flinn, Arushi Aggarwal
Summary
Introduced Owl, a new system for high-fanout distribution of large data objects (executables, code artifacts, AI models, search indexes) within Meta's private cloud. Owl addresses the distribution challenge by balancing fanout, size, and hotness requirements, improving download speeds and cache hit rates by 2-3x over prior systems like BitTorrent. It employs a split design with a decentralized data plane (peer libraries) and a centralized control plane (trackers) to manage distribution decisions, offering high customization through modular policies and trace-driven emulation.
More from the Scalability and Large-Scale Systems Engineering trail
- Open-Sourcing Rebalancer: A Generic, High-Performance Library for Solving Assignment Problems
- ZGateway: Learnings from Putting a Proxy in Front of ZippyDB
- MTIA 300: Meta’s First Training Chip with Built-in NICs and Communication-Offloading Engines
- From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta’s Ads Ranking
- GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model