Reinforcement learning: Why alignment of numerics and MoE routing matter

This post details the critical importance of numerical alignment between reinforcement learning (RL) training and inference for achieving stable and effective model updates, particularly with Mixture-of-Experts (MoE) models. It highlights how numerical mismatches can derail training by distorting probability updates and influencing MoE routing decisions. The post presents experimental evidence from GLM 5.2 and Qwen3.5-MoE, demonstrating that discrepancies in precision, kernel execution, and aggregation order can lead to collapsed rewards or incorrect expert selection. Fireworks' approach to solving this is by co-optimizing the trainer and rollout engine, implementing router replay for consistent expert selection, and validating agreement using training-inference KL divergence.