Cerebras-GPT: A Family of Open, Compute-efficient, Large Language Models - Cerebras

This post introduces Cerebras-GPT, a family of seven open-source GPT models (111M to 13B parameters) trained using the Chinchilla formula on Cerebras CS-2 systems. It establishes a new scaling law based on the public Pile dataset, predicting model performance for downstream tasks. The training leveraged Cerebras's data-parallel weight streaming architecture on a 16x CS-2 cluster, simplifying distributed training and enabling researchers to focus on ML design.