Efficient Large-Scale GPT Training Using a Cerebras Wafer-Scale Cluster - Cerebras

This post details the replication of pre-training a GPT-3 XL (1.3 billion parameters) model on the Pile dataset for 112B tokens using a Cerebras Wafer-Scale Cluster. It highlights the ease of scaling on an 8 x CS-2 cluster with a data-parallel weight streaming architecture and reports zero-shot evaluation accuracy across 24 downstream tasks, demonstrating comparable results to a state-of-the-art external baseline.