Accelerating Large GPT Training with Sparse Pre-Training and Dense Fine-Tuning [Updated] - Cerebras

This post introduces Sparse Pre-training and Dense Fine-tuning (SPDF) for large GPT models. It demonstrates that pre-training GPT-3 XL (1.3B parameters) with up to 75% unstructured sparsity on the Cerebras CS-2 system reduces training FLOPs by 60% compared to dense training. Subsequent dense fine-tuning preserves downstream task metrics on data-to-text NLG tasks (E2E, WebNLG, DART) with minimal BLEU score degradation (-0.44 to -0.75). The post also shows that simpler NLG tasks tolerate higher sparsity than complex summarization tasks (Curation Corpus), and larger models are more amenable to higher sparsity. The Cerebras CS-2's hardware acceleration for unstructured sparsity is highlighted as a key enabler for these large-scale sparsity studies.