How Decagon shipped real-time voice AI on Modal

This post details the collaboration between Decagon and Modal to achieve real-time voice AI inference. Key contributions include: 1. Training a custom EAGLE3 draft model using SpecForge and a specialized pipeline, resulting in a 38% higher accept length compared to open-source baselines. 2. Re-engineering the SGLang inference engine by building a new asynchronous scheduler for speculative decoding, eliminating host-side inefficiencies, enabling parallel GPU kernel execution, and achieving sync-free speculative decoding. This resulted in up to 12% throughput improvement and p90 latency of 342ms.