Unlocking Agentic RL Training for GPT-OSS: A Practical Retrospective

This post details the practical challenges and solutions encountered when enabling agentic RL training for the GPT-OSS model using the verl framework. Key contributions include: 1. Identifying and fixing a MoE log-probability mismatch that caused exploding KL divergence and non-increasing rewards in on-policy PPO training. 2. Implementing attention sink support in FlashAttentionV3 to handle the specific attention patterns required by GPT-OSS. 3. Introducing memory-efficient training techniques, specifically mitigating FSDP memory blow-ups caused by repeated MoE expert materialization and enabling sequence parallel with Flash Attention V3.