Accelerate RL rollouts by up to 50% with distribution-aware speculative decoding

What this post added

Introduces Distribution-aware speculative decoding (DAS) to accelerate Reinforcement Learning (RL) rollouts by up to 50%. DAS addresses the RL post-training rollout bottleneck by employing an adaptive suffix tree drafter that stays current with evolving model weights without retraining, and a length-aware scheduling strategy that balances inter-GPU load and allocates speculation budgets intra-GPU to neutralize stragglers. The system exploits prompt reuse inherent in RL training. Experimental results demonstrate significant rollout time reduction (up to 50% for math RL, ~25% for code RL) with no degradation in reward quality or model outputs.