FlashNorm Optimization

FlashNorm is a novel optimization technique for RMS normalization layers in transformer models. It achieves significant speedups by folding the normalization gain into projection weights offline and deferring scalar division. This allows for parallel execution of matrix and vector units, reducing idle time. The implementation involves custom CUDA kernels to leverage tensor and CUDA cores concurrently. A key challenge addressed was a race condition arising from implicit stream joins, where stale buffers were read due to unfinished computations. Explicit stream synchronization was implemented to resolve this.