To Bfloat or not to Bfloat? That is the Question! - Cerebras

This post details the benefits of using the bfloat16 data format for automatic mixed precision training of deep learning models, specifically GPT-style models. It explains how bfloat16, with its 8 exponent bits, offers a greater dynamic range than float16, leading to improved training throughput (18% increase observed for GPT-3 XL), reduced weight growth, and better evaluation scores. The post also highlights that bfloat16 is less prone to numerical instability (underflows, overflows) compared to float16 and can be a near drop-in replacement for float32, often negating the need for loss scaling. Configuration details for enabling bfloat16 in mixed precision mode on Cerebras CS-2 are provided.