Teaching flow-matching text-to-speech models with RL | Decagon

This post introduces Flow-DPO and Flow-GRPO, novel reinforcement learning methods adapted for flow-matching TTS models. It addresses the challenge of applying policy gradient methods (like DPO and GRPO) to deterministic flow-matching models by introducing techniques to create probabilistic transitions. Flow-DPO uses a relative flow loss to approximate policy-versus-reference margins, while Flow-GRPO converts the deterministic ODE into a stochastic differential equation (SDE) with score correction. The post also details robust reward design strategies to prevent shortcutting and improve control over speech attributes, presenting results that show improved reliability and pace control compared to previous methods.