MXFP8 Kernel Optimization for MoE Training

8 posts · Aug 2025 – Aug 2026

What this trail covers

Warp decode is an optimization for MoE inference on Blackwell GPUs that reorganizes parallelism around outputs rather than experts, achieving 1.84x throughput improvement and better accuracy by reducing bookkeeping steps, eliminating intermediate buffers, and leveraging warp independence for efficient scheduling. This accelerates the research and training pipeline for Composer. This post details the implementation of custom MXFP8 kernels for MoE training on Blackwell GPUs, achieving significant performance gains by optimizing dequantization and quantization processes, and overcoming hardware-specific challenges like TMEM latency.

All Cursor trails

Aug 2026Mixture-of-Kittens: our open-source MoE megakernel for NVL72sIntroduced Mixture-of-Kittens (MoK), a production MoE training megakernel for NVL72s. MoK fuses MoE communication and computation into a single, fully deterministic kernel, addressing the bottleneck of MoE layers in large model training. It achieves significant performance improvements by optimizing for NVL72 hardware, including overlapping computation and communication, choosing the right communication direction (push vs. pull), and minimizing CPU-GPU synchronization. MoK powers Composer training across tens of thousands of GPUs.
Apr 2026Speeding up GPU kernels by 38% with a multi-agent systemDetails a multi-agent system that autonomously optimized CUDA kernels, achieving a 38% geomean speedup by building and optimizing Blackwell GPU kernels from scratch. The system generated 235 optimization problems from production open-source models and achieved significant performance improvements, with 19% of optimizations exceeding 2x. Specific examples include BF16 Grouped Query Attention with Paged Prefill, NVFP4 MoE Linear with Gating, and BF16 Matrix Multiplication, showcasing the system's adaptability and ability to discover novel optimization strategies.
Apr 2026Better MoE model inference with warp decodeIntroduced 'warp decode', a novel approach to MoE inference kernel design that reorganizes parallelism around individual output values (neurons) rather than experts. This eliminates five bookkeeping stages present in conventional MoE paths, removes intermediate activation gather and per-expert output buffers, and leverages warp independence for improved GPU scheduling and latency hiding. Achieves 1.84x throughput gain and 1.4x closer accuracy to FP32 reference on Blackwell GPUs.
Mar 2026A technical report on Composer 2Details the training of Composer 2, a coding model for agentic software engineering. This includes continued pretraining on Kimi K2.5, large-scale reinforcement learning emulating the real Cursor environment, and evaluation using CursorBench. The post highlights performance improvements over Composer 1.5 and competitive results on public benchmarks, achieved at lower inference cost. It also describes the underlying infrastructure, including custom low-precision kernels for MoE training on Blackwell GPUs, an asynchronous RL pipeline, and Anyrun for running sandboxed coding environments.
Mar 2026Improving Composer through real-time RLIntroduced real-time Reinforcement Learning (RL) for improving the Composer AI model by using real inference tokens from user interactions as training signals. This enables rapid iteration with new checkpoints deployed every five hours. The process involves client-side instrumentation, backend data pipelines, and a fast deployment path. Addressed train-test mismatch by using real environments and users. Mitigated reward hacking by adapting the training to correctly handle invalid tool calls and stabilize editing behavior through reward function modifications. The post also details A/B testing results showing improvements in agent edit persistence, user dissatisfaction, and latency.
Advertisement
Mar 2026Introducing Composer 2Introduced Composer 2, a new frontier-level AI coding agent. This release focuses on improved intelligence and cost-effectiveness, with pricing at $0.50/M input and $2.50/M output tokens. A faster variant is also available. The post highlights performance gains on benchmarks like Terminal-Bench 2.0 and SWE-bench Multilingual, attributing these improvements to continued pretraining and reinforcement learning on long-horizon tasks.
Oct 2025Composer: Building a fast frontier model with RLIntroduced Composer, a new agent model for software engineering intelligence and speed, achieving frontier coding results with generation speed four times faster than similar models. Composer is a mixture-of-experts (MoE) language model specialized for software engineering through reinforcement learning (RL). Custom training infrastructure leveraging PyTorch and Ray was built for asynchronous RL at scale, natively training models at low precision by combining MXFP8 MoE kernels with expert parallelism and hybrid sharded data parallelism. Infrastructure for running hundreds of thousands of concurrent sandboxed coding environments for RL was adapted and rewritten.
Aug 20251.5x faster MoE training with custom MXFP8 kernelsThis post details the implementation of custom MXFP8 kernels for Mixture-of-Experts (MoE) training on Blackwell GPUs. It addresses challenges related to tensor memory (TMEM) latency and the dominance of quantization overhead. The authors rewrote the MoE layer from scratch using pure CUDA and PTX, achieving a 3.5x improvement in MoE layer performance and a 1.5x end-to-end training speedup on Blackwell GPUs. Key contributions include a detailed explanation of MXFP8 quantization, analysis of dequantization bottlenecks on Blackwell, and a recipe for efficient FP8 quantization kernels.