How Together AI built the world’s fastest speech-to-text stack

This post details engineering changes to achieve world-leading ASR performance, including TensorRT multi-profile engines for encoder optimization, conditional CUDA graphs for GPU-side decoder control flow, lower-copy CPU paths using persistent Unix domain sockets and shared memory, evented streaming I/O with epoll for reduced scheduler pressure, and runtime GC control via `gc.freeze()` to eliminate tail latency spikes.