How Together AI built the world’s fastest speech-to-text stack

What this post added

This post details the technical optimizations implemented to achieve industry-leading speech-to-text performance. Key contributions include: 1. TensorRT multi-profile engines for the encoder to handle variable audio shapes efficiently. 2. Conditional CUDA graphs to move the decoder's conditional branching onto the GPU, eliminating CPU round trips. 3. Collapsing CPU preprocessing into fewer processes and using custom protocols over Unix domain sockets or shared memory for lower-copy data paths. 4. Implementing evented I/O with `epoll` for streaming to reduce scheduler pressure and improve predictability. 5. Utilizing `gc.freeze()` to prevent Python's garbage collector from causing latency spikes in streaming workflows.