Variable Sequence Length Training for Long-Context Large Language Models - Cerebras

Introduces Variable Sequence Length (VSL) training for LLMs. VSL stages training into short (e.g., 2k) and long (e.g., 8k) sequence phases. This method reduces FLOPs by up to 29% compared to training with long sequences throughout, without requiring architectural changes or hyperparameter tuning. It demonstrates comparable model performance and generalizes across different position embeddings (NoPE, learned, RoPE, ALiBi). The Cerebras Software Platform (CSoft) simplifies VSL implementation by modifying data subset paths.