Owl: Distributing content at Meta scale

What this post added

Introduced Owl, a new system for high-fanout distribution of large data objects (executables, code artifacts, AI models, search indexes) within Meta's private cloud. Owl addresses the challenges of fanout, size, and hotness of content distribution by employing a decentralized data plane and a centralized control plane. This split design allows for efficient streaming of data from sources to clients via ephemeral, per-data-chunk distribution trees. Owl has scaled to distribute over 700 petabytes of data per day to over 10 million clients, improving download speeds and cache hit rates by 2-3x compared to previous systems like BitTorrent. The system emphasizes customizability through modular tracker interfaces for defining caching and fetching policies, and uses trace-driven emulation for policy optimization.