TechPrufer
Get started
NVIDIA
NVIDIA logoNVIDIA · Feature trail

Dense vs. Mixture-of-Experts (MoE) Model Architectures

2 posts · Jun 2026 – Sep 2026

What this trail covers

This feature thread tracks the evolution of understanding and deploying different AI model architectures, specifically comparing Dense models with Mixture-of-Experts (MoE) models. Initial efforts focused on the fundamental differences in parameter activation and their impact on performance. This post details the introduction of advanced fused MLP kernels for MoE models, custom-built with the NVIDIA CuTe DSL. These kernels address memory and synchronization bottlenecks, enabling sync-free MoE execution. Subsequent developments have introduced TE's GroupedLinear for reduced kernel launch overhead, MXFP8 precision for memory reduction, and a fused GroupedMLP kernel that combines quantization, SwiGLU activation, and routing-weight scaling, significantly improving training throughput for MoE-based biological foundation models.

All NVIDIA trails
  1. Efficient MoE Training for Biological Foundation Models | NVIDIA Technical BlogThis post details the implementation of NVIDIA Transformer Engine (TE) optimizations for training Mixture-of-Experts (MoE) models, specifically within the BioNeMo recipe for biological foundation models. Key contributions include: 1. Using TE's GroupedLinear to reduce kernel launch overhead by submitting multiple expert GEMMs as a single grouped operation, contrasting with naive Python loop-based expert iteration. 2. Leveraging MXFP8 precision on NVIDIA Blackwell GPUs to reduce memory usage compared to BF16, with hardware acceleration for MXFP8 GEMMs. 3. Introducing a fused GroupedMLP kernel via the TE Sequential API that combines GroupedLinear, ScaledSwiGLU, and routing-weight scaling into a single operation, avoiding intermediate materialization and reducing framework overhead. Benchmarks show up to 2.21x throughput improvement over a Hugging Face baseline.
  2. Dense vs. MoE Models: Active Parameters, Throughput, and When to Choose Each | NVIDIA Technical BlogThis post elaborates on the trade-offs between Dense and Mixture-of-Experts (MoE) model architectures. It explains that MoE models activate only a subset of parameters per token, decoupling memory cost from compute cost, which leads to higher token throughput compared to dense models at equal total parameter counts. The post also details how MoE models can narrow their latency advantage at high concurrency due to routing and memory movement overhead. It discusses the impact of quantization on MoE components and emphasizes that deployment choice should be based on memory budget, concurrency needs, fine-tuning plans, and quantization behavior, rather than solely on raw parameter count. The Nemotron 3.5 Lightning model is used as a case study, highlighting its hybrid MoE architecture with Mamba-2 layers and speculative decoding.
  3. Accelerating Dropless MoE Training in JAX with NVIDIA Transformer Engine | NVIDIA Technical BlogThis post details optimizations for dropless MoE training in JAX using NVIDIA Transformer Engine and NCCL EP. Key contributions include: 1) Grouped GEMM kernels that handle variable expert token counts natively, avoiding padding and improving Tensor Core utilization, with MXFP8 support on Blackwell GPUs. 2) Optimized Expert Parallelism (EP) operations that fuse dispatch and combine stages, reducing network traffic and latency through token deduplication and efficient all-to-all communication. 3) Additional optimizations like JAX host offloading and XLA multistreaming collectives to mitigate memory bottlenecks and overlap communication. These advancements resulted in a 10.4x throughput improvement for DeepSeek-V3 training on NVIDIA GB200 and sustained 97% scaling efficiency at 1,024 GPUs on NVIDIA GB300 NVL72.
  4. Experiment with Qwen3.8-Flash-Next on NVIDIA GB300 NVL72 for Agentic Coding | NVIDIA Technical BlogThis post details the Qwen3.8-Flash-Next model, a multimodal MoE model with 125B parameters and 6B activated per token, featuring a native 262,144-token context window extensible to 1M tokens. It introduces a hybrid architecture combining Gated DeltaNet (GDN) for context compression into a recurrent state and Qwen Sparse Attention (QSA) for precise retrieval. QSA aggregates sequences into micro-blocks to reduce attention, compute, and indexing overhead, achieving up to 7.6x prefill and 4.9x decoding speedups over full attention for 1M-token workloads. The post also presents performance benchmarks on NVIDIA GB300 NVL72, showing over 16K tokens/sec/GPU and 200 tokens/sec/user, and mentions fine-tuning with NVIDIA NeMo AutoModel and RL recipes, as well as inference support from SGLang, vLLM, and TokenSpeed.
  5. Serve Qwen3.8-2.4T-A95B, a 2.4T-Parameter Model, with Configurable Reasoning on NVIDIA GB300 NVL72 | NVIDIA Technical BlogThis post details the deployment of Qwen3.8-2.4T-A95B, a 2.4T-parameter model, on NVIDIA GB300 NVL72. It highlights the model's hybrid full/linear attention architecture and fine-grained MoE design for efficient long-context inference. The post quantifies performance metrics, achieving over 4K tokens/sec/GPU and 350 tokens/sec/user in FP8 precision. It also discusses serving options via SGLang, vLLM, NVIDIA Dynamo, and NVIDIA NIM, and fine-tuning with NVIDIA NeMo AutoModel.
  7. Boosting MoE Training Throughput with Advanced Fusion Kernels | NVIDIA Technical BlogIntroduced advanced fused MLP kernels for MoE models using the NVIDIA CuTe DSL. These kernels eliminate memory and synchronization overhead by fusing GroupGemm with quantization and activation functions (SwiGLU, GeGLU, sReLU). They enable sync-free MoE execution with full-iteration CUDA graphs, removing CPU synchronization bottlenecks. The kernels natively handle MXFP8 and NVFP4 quantization, feature scaling, clamping, and bias addition. This results in 1.3x-2x kernel-level speedup and contributes to significant end-to-end throughput improvements in pre-training setups.
  8. Deploy Long-Context Reasoning and Agentic Workflows with MiniMax M3 on NVIDIA Accelerated Infrastructure | NVIDIA Technical BlogThis post introduces the MiniMax M3 model, a 428B parameter Mixture-of-Experts (MoE) model with 1M-token context and native multimodality, deployed on NVIDIA Blackwell infrastructure. It highlights the MiniMax Sparse Attention (MSA) mechanism, which replaces standard quadratic attention with a pre-filtering stage for more efficient KV cache access, resulting in over 4x faster contiguous KV cache reads, 1/20th per-token compute cost at 1M context, and significant speedups in prefill and decoding without precision loss. The post details deployment options using TensorRT LLM, SGLang, and vLLM, and discusses scaling with NVIDIA Dynamo for large-scale applications. It also covers customization and fine-tuning using the NVIDIA NeMo Framework, including support for SFT, LoRA, and reinforcement learning, with context parallel support up to 128k tokens.
  9. NVIDIA Nemotron 3 Ultra Powers Faster, More Efficient Reasoning for Long-Running Agents | NVIDIA Technical BlogThis post introduces NVIDIA Nemotron 3 Ultra, a 550B-parameter Mixture-of-Experts (MoE) model with 55B active parameters, optimized for long-running agent workflows. Key technical contributions include: architectural innovations such as hybrid Mamba-Transformer layers for efficient long-context handling and precise recall, NVFP4 quantization for cross-architecture GPU deployment with up to 5x higher throughput, LatentMoE for more efficient expert routing, and multi-token prediction (MTP) to reduce generation time. It also details the Multi-Teacher On-Policy Distillation (MOPD) training method, which uses specialized teacher models to improve reasoning across domains, and highlights the release of extensive training data, including 212B new tokens targeting legal, Wiki-based, and GitHub data, along with new SFT samples, RL tasks, and RL environments.