Cerebras

This post provides a detailed breakdown of Mixture-of-Experts (MoE) inference arithmetic, clarifying the meaning of terms like '8x7B' and explaining the actual active parameters. It offers in-depth calculations for memory requirements (model weights and KV-cache) and FLOPs (prefill and decode stages) for MoE models. The post contrasts deployment bottlenecks across different hardware setups: single-GPU (memory-bound), multi-GPU (communication-bound), and specialized hardware like Cerebras WSE (compute-bound). It introduces the concept of KV-caching to make decode stage compute linear with sequence length and explains the trade-offs involved.