Parcae: Doing more with fewer parameters using stable looped models

What this post added

This post introduces Parcae, a novel stable architecture for looped language models. It details the challenges in training prior looped models (instability, state explosion, loss spikes) and presents a theoretical framework based on linear dynamical systems to understand and address these issues. Parcae achieves stability by parameterizing input injection and constraining the spectral radius of the update matrix. The post empirically demonstrates Parcae's ability to match the quality of larger Transformers with fewer parameters, establishes new scaling laws for looping, and provides training code and models for community use.