Scaling LLM Inference: Innovations in Tensor Parallelism, Context Parallelism, and Expert Parallelism

This post details innovations in LLM inference parallelism, specifically Tensor Parallelism (TP) with Direct Data Access (DDA) algorithms (flat and tree variants) to reduce allreduce latency, Context Parallelism (CP) for long-context models (e.g., 1M/10M tokens) using 'ring attention' variants (Pass-KV, Pass-Q), and Expert Parallelism (EP) for MoE models with optimizations for all-to-all communication. It also outlines future directions like N-D parallelism and disaggregated prefill/decoding tiers.