TechPrufer
Get started
Decagon
Decagon logoDecagon · Feature trail

Real-time Text-to-Speech Inference Scaling

8 posts · Aug 2026 – Sep 2026

What this trail covers

Decagon has significantly scaled its real-time text-to-speech (TTS) inference by rebuilding the pipeline on the M* modular inference framework. This involved stage decomposition, packed inputs, cache and batch-state integration, causal decoder history management, CUDA Graphs, and validation. The new approach enables stage-aware scheduling and true tensor batching, leading to higher capacity and reduced latency. Key improvements include packed prefill for variable-length prompts, paged KV caching. Decagon has now developed Chord, a post-trained speech model specifically for customer conversations, which improves naturalness, engagement, and resolution rates. Chord utilizes a dedicated pipeline for conversational data, verbatim transcription, and a tokenizer-free diffusion model for enhanced steerability and detail preservation. Production deployment is handled via Modal.

All Decagon trails
  1. How we made one of our largest inference workloads 4.7× more GPU-efficient | DecagonThis post details optimizations for a scheduled inference workload, achieving a 4.7x improvement in fleet-level GPU efficiency. Key contributions include disaggregating prompt processing from token generation to allow for specialized batching and state transfer, implementing an admission controller to manage traffic bursts and ensure redistributability of requests, and reducing worker startup time by prefetching model files. These changes address bottlenecks in batch formation, memory pressure, state transfer, burst admission, and worker startup.
  2. Scaling real-time text-to-speech inference | DecagonThis post details the technical implementation of scaling real-time text-to-speech inference by rebuilding the pipeline on the M* modular inference framework. It introduces stage decomposition (packed prefill, latent generation, waveform decoding), paged KV caching, continuous batching of ready work, isolating causal decoder history, and chaining context across clauses. The use of CUDA Graphs for replaying common GPU operations is also discussed, along with the tradeoffs involved in capturing different batch sizes and prompt lengths. Performance metrics show significant improvements in TTFA, end-to-end latency, RTF, audio throughput, and request throughput compared to sequential and interleaved approaches, and outperform vLLM-Omni.
  3. Teaching flow-matching text-to-speech models with RL | DecagonThis post introduces Flow-DPO and Flow-GRPO, novel reinforcement learning methods adapted for flow-matching TTS models. It addresses the challenge of applying policy gradient methods (like DPO and GRPO) to deterministic flow-matching models by introducing techniques to create probabilistic transitions. Flow-DPO uses a relative flow loss to approximate policy-versus-reference margins, while Flow-GRPO converts the deterministic ODE into a stochastic differential equation (SDE) with score correction. The post also details robust reward design strategies to prevent shortcutting and improve control over speech attributes, presenting results that show improved reliability and pace control compared to previous methods.
  4. How Decagon shipped real-time voice AI on Modal | DecagonThis post details the collaboration with Modal to achieve real-time voice AI inference. Key contributions include: 1. Training a custom speculative draft model (forking SpecForge) that achieved 38% higher accept lengths than open-source baselines. 2. Re-engineering SGLang by building a new asynchronous scheduler for speculative decoding, eliminating CPU stalls and host-side inefficiencies, enabling parallel GPU kernel execution, and improving throughput by up to 12%.
  5. UndatedIntroducing Chord: a speech model built for customer conversations | DecagonDecagon developed Chord, a new speech model post-trained specifically for customer conversations. This involved creating a dedicated pipeline for conversational data that preserves natural disfluencies and pacing, using verbatim transcription and a novel recording method for voice talent. Chord is based on a tokenizer-free diffusion model to maintain audio fidelity and improve steerability for emotion, pace, and emphasis. Production deployment is handled on Modal. The model demonstrated improvements in resolution rate (6.1-7.1 points) and reduced barge rates (5.3-14.5 points) across industries, and was found to be nearly indistinguishable from human voices in blind tests.
  7. UndatedIntroducing Voice 3 and Chord, our new speech model | DecagonThis post introduces Voice 3, featuring the Chord speech model and a new duplex architecture. Chord is a voice model specifically trained and post-trained on real-world CX conversations, shaping speech phrase by phrase for naturalness and conversational pace. The duplex architecture separates a low-latency conversational model (listening/speaking) from a more powerful reasoning model (decision-making, tool-calling), enabling simultaneous operations and more natural turn-taking. The system supports 70+ languages with automatic language switching and locale-specific voices.