Dense vs. MoE Models: Active Parameters, Throughput, and When to Choose Each | NVIDIA Technical Blog

This post elaborates on the trade-offs between Dense and Mixture-of-Experts (MoE) model architectures. It explains that MoE models activate only a subset of parameters per token, decoupling memory cost from compute cost, which leads to higher token throughput compared to dense models at equal total parameter counts. The post also details how MoE models can narrow their latency advantage at high concurrency due to routing and memory movement overhead. It discusses the impact of quantization on MoE components and emphasizes that deployment choice should be based on memory budget, concurrency needs, fine-tuning plans, and quantization behavior, rather than solely on raw parameter count. The Nemotron 3.5 Lightning model is used as a case study, highlighting its hybrid MoE architecture with Mamba-2 layers and speculative decoding.