Weight Folding, CUDA Streams, and the Bug That Made My Model Speak Backwards — Filip Makraduli

This post details the implementation of FlashNorm, an optimization for RMS normalization layers. It describes the algebraic simplification of folding gain into projection weights and deferring division. The core technical contribution is the custom CUDA kernel that parallelizes matmul on tensor cores and RMS reduction on CUDA cores. It also highlights a critical bug discovered during generation where a race condition caused the model to output stale data due to an implicit join between CUDA streams. The fix involved explicit stream synchronization, ensuring the post-scale operation waits for all preceding computations to complete.