Cerebras

This post analyzes three fundamental MoE routing strategies: Hash Routing, Learned Routing, and Sinkhorn Routing. It details their mechanisms, performance tradeoffs (expert utilization vs. specialization), and practical implementation challenges. Learned Routing is highlighted as the industry standard due to its performance, while acknowledging its router collapse issue. Sinkhorn Routing is presented as a method to achieve per-layer load balancing but with scaling and gradient detachment complexities. The post also discusses common engineering tricks layered on these core methods, such as capacity factors and auxiliary loss, and references specific models like DeepSeek-V3 and Qwen2 for their routing implementations. It concludes by emphasizing the fragility of MoE training and the importance of understanding these core routing mechanisms for debugging.