Creating Sparse GPT-3 Models with Iterative Pruning - Cerebras

Introduced iterative pruning to create highly sparse GPT-3 models (1.3B parameters) on the Cerebras CS-2 system. Demonstrated an 83.8% sparse model with a 3x reduction in inference FLOPs and 4.3x reduction in parameters, while maintaining competitive validation loss on the Pile dataset and matching zero-shot performance on LAMBADA and WikiText103 compared to the dense model. The method sparsifies projection and feed-forward network layers separately and involves retraining after pruning.