Toward RL on mixed-precision, mixed-accelerator samplers

This post details the implementation of mixed-precision, mixed-accelerator samplers for large-model reinforcement learning. It introduces off-policy correction techniques (Masked Importance Sampling, Score Centering, and Top-p sampling with mask replay) to stabilize training despite training-inference mismatch. The system leverages Modal's autoscaling, serverless GPUs to manage diverse samplers and demonstrates achieving performance on par with homogeneous samplers at lower cost, specifically for agentic coding tasks.