Cerebras

What this post added

Introduced REAP (Router-weighted Expert Activation Pruning), a one-shot pruning method for MoE models. REAP measures expert importance by combining router gate-values and expert output magnitude to identify and remove low-impact experts. Demonstrated that pruning is superior to merging for generative tasks due to the 'functional subspace collapse' issue caused by merging. Showcased REAP's ability to prune up to 50% of experts in trillion-parameter models while maintaining high model quality across various generative benchmarks.