Cerebras

Introduces a practical implementation of Mixture-of-Experts (MoE) models using a modified nanogpt codebase. Details the architecture of MoE, including expert networks and mixing logic. Explains the implementation of learned routing with auxiliary loss for load balancing. Identifies and diagnoses a critical bug in learned routing with top_k=1 where the router receives zero gradient from the cross-entropy loss, leading to ineffective expert selection. Proposes and implements a 'null expert' solution to boost gradient flow and improve router performance.