Why off-policy training isn't enough: SFT, RL, and the limits of imitation | Decagon

What this post added

This post details the limitations of Supervised Fine-Tuning (SFT) for AI agent conversation classification, specifically the issues of compounding errors at inference, the ineffectiveness of self-distillation, and the erosion of steerability. It introduces 'steerability' as a critical metric, measured by a benchmark that quantifies a model's response to perturbations in category definitions. The post then discusses on-policy distillation as an improvement over SFT by closing the train-inference gap, but notes its dependency on teacher quality. Finally, it advocates for Reinforcement Learning (RL) with verifiable rewards as the most robust approach, enabling models to generalize beyond teacher supervision and discover correct reasoning strategies. The optimal training progression is presented as SFT -> on-policy distillation -> RL, each stage addressing the shortcomings of the previous one to create a more robust and correctable AI agent.