Cerebras

The post details the training and deployment of Jais 2, a family of Arabic-centric LLMs, on Cerebras wafer-scale clusters. It highlights the achievement of 2,000 tokens per second inference speed for the Jais 2 70B model, attributed to Cerebras's on-wafer SRAM architecture and speculative decoding stack. The training process utilized a unified MemoryX and SwarmX fabric, enabling weight streaming and eliminating the need for complex parallelism strategies, leading to near-linear performance scaling. The post also discusses the model development pipeline, including pretraining on 2.6 trillion tokens, supervised fine-tuning, and Direct Preference Optimization (DPO) for alignment.