The Practitioner’s Guide to the Maximal Update Parameterization - Cerebras

This post introduces and explains Maximal Update Parameterization (μP) as a technique to improve AI model training. It details four key benefits: stable hyperparameters across scale (μTransfer), improved loss at large scale due to better HP tuning, significantly decreased danger of instability at large scale, and more predictable scaling. The post provides a mathematical overview of controlling activation magnitudes, gradients, and weight updates to ensure these dynamics are invariant to model width. It also highlights a practical implementation port to nanoGPT.