Why off-policy training isn't enough: SFT, RL, and the limits of imitation | Decagon

This post details a novel training progression for AI classification models, moving beyond standard Supervised Fine-Tuning (SFT). It introduces a sequential approach combining SFT, on-policy distillation, and Reinforcement Learning (RL) to address limitations in model robustness and steerability. Key contributions include: 1) identifying SFT's structural limitations (compounding errors, steerability erosion), 2) proposing a steerability benchmark to quantify these failure modes, 3) demonstrating that on-policy distillation improves accuracy and steerability but introduces teacher dependency, and 4) showing that RL with verifiable rewards, initialized with SFT, leads to significantly more robust models that are correctable via prompt updates. The post also references prior work on fine-tuning AI agents and knowledge distillation.