Scaling Up and Out: Training Massive Models on Cerebras Systems using Weight Streaming - Cerebras

This post introduces and details the 'weight streaming' execution mode for the Cerebras Wafer-Scale Engine (WSE), enabling the training of AI models with over 100 trillion parameters, far exceeding the WSE's on-chip memory capacity. It explains how model parameters are managed in an off-chip memory tier with 4 TB/s bandwidth, while the WSE continuously processes streams of weights and gradients. The post also elaborates on the WSE's architecture, its ability to leverage 90% unstructured weight sparsity for a 10x acceleration, and its full-performance execution of all linear algebra operations (GEMM, GEMV, vector-vector) which is critical for both training and low-latency inference.