Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta

This post details Meta's approach to operating distributed inference systems at scale, highlighting the increasing complexity and coupling between different layers (routing, KV cache, batching, autoscaling). It introduces the concept of a request as a distributed transaction and the need for a dedicated control plane, analogous to Kubernetes for virtual machines. The discussion covers seven axes for an inference scheduler (GPU generation, memory headroom, KV cache state, weight warmth, tenant priority, latency budget, workflow context) and four optimization quadrants (avoid, share, move, delay). The primary metric is cost per successful task, and observability is framed as a control loop input to balance latency, cost, and throughput.