Scaling real-time text-to-speech inference | Decagon

This post details the technical implementation of scaling real-time text-to-speech inference by rebuilding the pipeline on the M* modular inference framework. It introduces stage decomposition (packed prefill, latent generation, waveform decoding), paged KV caching, continuous batching of ready work, isolating causal decoder history, and chaining context across clauses. The use of CUDA Graphs for replaying common GPU operations is also discussed, along with the tradeoffs involved in capturing different batch sizes and prompt lengths. Performance metrics show significant improvements in TTFA, end-to-end latency, RTF, audio throughput, and request throughput compared to sequential and interleaved approaches, and outperform vLLM-Omni.