TechPrufer
Get started
NVIDIA
NVIDIA logoNVIDIA · Feature trail

Kernel Fusion in CUDA

2 posts · Jun 2026 – Jul 2026

What this trail covers

This feature thread tracks the evolution of kernel fusion techniques in NVIDIA CUDA to optimize GPU performance. Initial efforts focused on manual kernel fusion in CUDA C++ to reduce memory traffic and kernel launch overhead by combining multiple operations into a single kernel. Subsequent developments have explored implicit kernel fusion through compilers like PyTorch's Torch Inductor, which automatically fuses operations like element-wise transformations and reductions. This post details how custom fused MLP kernels, built with the CuTe DSL, address MoE bottlenecks by eliminating memory and synchronization overhead, enabling sync-free MoE execution with full-iteration CUDA graphs and achieving 1.3x-2x kernel-level speedup over unfused paths. These fused kernels support advanced GLU activation functions and quantization, and integrate with cuDNN Frontend, Transformer Engine, and Megatron Core, delivering significant end-to-end throughput improvements in pre-training setups.

All NVIDIA trails
  1. Kernel Fusion in NVIDIA CUDA: Optimizing Memory Traffic and Launch Overhead | NVIDIA Technical BlogThis post details how kernel fusion can improve memory bandwidth and reduce kernel launch overhead by combining multiple GPU operations into a single device kernel. It provides examples of manual kernel fusion in CUDA C++ and implicit kernel fusion using PyTorch's `torch.compile`, demonstrating performance improvements and reduced memory traffic.
  2. Boosting MoE Training Throughput with Advanced Fusion Kernels | NVIDIA Technical BlogIntroduces custom fused MLP kernels for MoE models, built with the CuTe DSL, to address activation, CPU boundedness, and quantization costs. These kernels achieve 1.3x-2x kernel-level speedup by enabling sync-free MoE execution with full-iteration CUDA graphs. They support GLU activation functions (SwiGLU, GeGLU, sReLU) with fused epilogues, handle quantization (MXFP8, NVFP4) and other operations like clamping and bias addition within the fused kernel, and eliminate CPU synchronization overhead by tracking tokens per group within GPU memory. Integration with cuDNN Frontend, Transformer Engine, and Megatron Core leads to up to 8% end-to-end throughput improvement in DeepSeek-V3 and 93% in GPT-OSS pre-training.