Differential Transformer V2

This post introduces Differential Transformer V2 (DIFF V2), an evolution of DIFF V1. DIFF V2 focuses on inference efficiency by matching baseline Transformer decoding speed without custom kernels, and improving training stability by removing per-head RMSNorm, which was found to cause instability in large-scale pretraining. It also simplifies parameterization and initialization. Experiments on production-scale LLMs show lower language modeling loss, reduced gradient spikes, and fewer activation outliers compared to standard Transformers. DIFF V2 also addresses the Softmax constraint and helps eliminate attention sinks.