Extending LLM context with 99% less training tokens - Cerebras

This post presents a new recipe for extending LLM context length with significantly reduced training tokens (~99% reduction compared to full context training). It details the use of Retrieval Augmented Fine-Tuning (RAFT) with distractor passages, syntactic question generation, and translation for data augmentation. It also incorporates train-time token position ID shifting (simplified CREAM) and RoPE position extrapolation with an adjusted base frequency. The training pipeline consists of CPT@32K, SFT@8K, and SFT@32K stages, with specific data mixes and hyper-parameter settings provided for each.