Parcae: Doing more with fewer parameters using stable looped models

This post introduces Parcae, a stable architecture for looped language models that achieves the quality of a Transformer twice its size with predictable training. It addresses instability issues in prior looped models by analyzing their linear dynamical system properties and constraining the spectral radius of the recurrent block's update matrix. Parcae demonstrates improved validation perplexity and downstream task performance compared to parameter- and data-matched Transformers and prior looped models (RDMs). The post also establishes the first scaling laws for looping, showing that compute-optimal training requires increasing looping and data in tandem, and that looping creates a stricter Pareto Frontier for validation loss. Training code and models for Parcae are released.