How Decagon shipped real-time voice AI on Modal | Decagon
Aug 13, 2026
Summary
This post details the collaboration with Modal to achieve real-time voice AI inference. Key contributions include: 1. Training a custom speculative draft model (forking SpecForge) that achieved 38% higher accept lengths than open-source baselines. 2. Re-engineering SGLang by building a new asynchronous scheduler for speculative decoding, eliminating CPU stalls and host-side inefficiencies, enabling parallel GPU kernel execution, and improving throughput by up to 12%.
More from the Real-time Text-to-Speech Inference Scaling trail
- How we made one of our largest inference workloads 4.7× more GPU-efficient | Decagon
- Scaling real-time text-to-speech inference | Decagon
- Teaching flow-matching text-to-speech models with RL | Decagon
- Introducing Voice 3 and Chord, our new speech model | Decagon
- Introducing Chord: a speech model built for customer conversations | Decagon