BTLM-3B-8K: 7B Performance in a 3 Billion Parameter Model - Cerebras

Introduces BTLM-3B-8K, a 3 billion parameter language model that achieves 7 billion parameter performance, optimized for long sequence length inference (8K+ tokens). It demonstrates the ability to run on devices with as little as 3GB of memory when quantized to 4-bit. The model was trained on the SlimPajama dataset and utilizes ALiBi position embeddings with variable sequence length training for improved extrapolation. The training process incorporated maximal update parameterization, improved hyperparameter tuning, an updated model architecture, extensive data cleaning and deduplication, and variable sequence length training. The post highlights the training on the CG-1 AI supercomputer, emphasizing the Cerebras weight streaming architecture's scalability and simplicity, and notes zero hardware failures during training.