TechPrufer
Sign inGet started

Blogs›NVIDIA›Kernel Fusion in CUDA

Kernel Fusion in CUDA

Kernel Fusion in CUDA

2
posts
2026

This feature thread tracks the evolution of kernel fusion techniques in NVIDIA CUDA to optimize GPU performance. Initial efforts focused on manual kernel fusion in CUDA C++ to reduce memory traffic and kernel launch overhead by combining multiple operations into a single kernel. Subsequent developments have explored implicit kernel fusion through compilers like PyTorch's Torch Inductor, which automatically fuses operations like element-wise transformations and reductions. This thread now includes advanced fused MLP kernels for MoE models, custom-built with the NVIDIA CuTe DSL, which eliminate memory and synchronization bottlenecks, enabling sync-free MoE execution for full-iteration CUDA graphs and delivering significant kernel-level speedups. These fused kernels support advanced GLU activation functions and natively handle quantization (MXFP8, NVFP4), feature scaling, tensor clamping, and bias addition, leading to end-to-end throughput improvements in large-scale pre-training setups.

2026

Kernel Fusion in NVIDIA CUDA: Optimizing Memory Traffic and Launch Overhead | NVIDIA Technical Blog

7/10/2026

This post introduces kernel fusion in NVIDIA CUDA as a technique to optimize memory traffic and launch overhead. It demonstrates manual kernel fusion by rewriting two separate kernels (abs and sum) into a single `sum_abs_kernel`, eliminating intermediate buffers and achieving a 3x speedup. It also shows how `torch.compile` with Torch Inductor implicitly fuses operations like `abs().sum()` by generating kernels where the absolute value is computed in registers before accumulation, avoiding separate kernels and intermediate buffers.

Boosting MoE Training Throughput with Advanced Fusion Kernels | NVIDIA Technical Blog

6/15/2026

Introduced advanced fused MLP kernels for MoE models, custom-built with the NVIDIA CuTe DSL, to address activation bottlenecks, CPU boundedness, and quantization costs. These kernels achieve 1.3x-2x kernel-level speedup over unfused paths by eliminating memory and synchronization overhead, enabling sync-free MoE execution with full-iteration CUDA Graphs. They support GLU activation functions (SwiGLU, GeGLU, sReLU) by repacking weights for in-epilogue computation and natively handle quantization (MXFP8, NVFP4), feature scaling, tensor clamping, and bias addition. Integration with cuDNN Frontend, Transformer Engine, and Megatron Core leads to up to 8% end-to-end throughput improvement in DeepSeek-V3 and 93% in GPT-OSS pre-training.