IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretraining

This post proposes an integrated enlarge-and-prune pipeline for generative language model pretraining, combining enlarged model training, pruning, and recovery under a single cosine annealing learning rate schedule. It also introduces an iterative structured pruning method for gradual parameter removal to mitigate knowledge loss and enable effective redistribution of model capacity, leading to superior performance of pruned models.