Turning Opportunistic AI Inference Capacity Into Better Ads Recommendations
This post details a new system designed to leverage idle AI inference capacity for ads recommendations. It introduces two elasticity levers, Rank More Ads (RMA) and Serve Better Models (SBM), to adapt to volatile and preemptible opportunistic capacity. RMA increases the number of ad candidates evaluated, requiring CPU and accelerator surplus, while SBM adjusts the share of requests routed to more powerful models, utilizing accelerator-only surplus. The system employs physical and logical isolation to protect baseline workloads, closed feedback loops with PID controllers for RMA latency and value-aware routing for SBM, and multi-layer value-aware load shedding to gracefully degrade when capacity is reclaimed. This system drives significant advertiser value and capacity efficiency by putting otherwise idle capacity to work.
More from the Scalability and Large-Scale Systems Engineering trail
- Open-Sourcing Rebalancer: A Generic, High-Performance Library for Solving Assignment Problems
- ZGateway: Learnings from Putting a Proxy in Front of ZippyDB
- MTIA 300: Meta’s First Training Chip with Built-in NICs and Communication-Offloading Engines
- From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta’s Ads Ranking
- GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model