Linear Scaling Made Possible with Weight Streaming - Cerebras

Introduces weight streaming via MemoryX and SwarmX, enabling linear scaling of LLMs up to trillions of parameters using only data parallelism. This disaggregates memory and compute, simplifying distributed compute challenges and offering push-button allocation of work. The Cerebras Wafer-Scale Cluster can scale from a single CS-2 to 192 CS-2s with a single keystroke. The WSE-2's large size allows fitting large layers onto a single device, and MemoryX provides unbounded storage for parameters and optimizer states by streaming weights to the CS-2 and gradients back. This allows a single CS-2 to train models up to 10 billion parameters, and larger models for fine-tuning or development.