Scaling LLM Inference: Innovations in Tensor Parallelism, Context Parallelism, and Expert Parallelism

What this post added

This post details Meta's advancements in scaling LLM inference by implementing and optimizing tensor parallelism (TP), context parallelism (CP), and expert parallelism (EP). It introduces direct data access (DDA) algorithms (flat and tree) to reduce allreduce latency in TP, achieving significant speedups over NCCL/RCCL. For CP, it describes 'ring attention' variants (Pass-KV, Pass-Q) enabling efficient long-context processing, achieving sub-minute inference for 1M tokens. For EP, it discusses optimizations for MoE models, including dynamic and persistent all-to-all communication. The post also outlines future directions like N-D parallelism and disaggregated inference tiers.