Improving Composer through real-time RL

Introduced real-time Reinforcement Learning (RL) for improving the Composer AI model by using real inference tokens from user interactions as training signals. This enables rapid iteration with new checkpoints deployed every five hours. The process involves client-side instrumentation, backend data pipelines, and a fast deployment path. Addressed train-test mismatch by using real environments and users. Mitigated reward hacking by adapting the training to correctly handle invalid tool calls and stabilize editing behavior through reward function modifications. The post also details A/B testing results showing improvements in agent edit persistence, user dissatisfaction, and latency.