Announcing the fastest inference for realtime voice AI agents

This post details the engineering behind Together AI's expanded voice infrastructure, focusing on real-time performance and developer experience. Key contributions include: 1. **Streaming Whisper STT:** Optimized model inference and system design (WebSocket streaming, tuned VAD) for real-time audio processing, achieving faster transcript completion (up to 35% faster than alternatives) by prioritizing time-to-complete-transcript. 2. **Serverless Open-Source TTS:** Introduction of Orpheus (187ms TTFB, natural expressiveness) and Kokoro (97ms TTFB, ultra-low latency) available via REST and WebSocket, eliminating the need for dedicated instances for sporadic TTS needs. 3. **Voxtral Mini and Speaker Diarization:** Integration of a higher-accuracy multilingual transcription model (Voxtral Mini) with measurably lower word error rates, and speaker diarization for structured conversation analysis. Architectural decisions emphasize low latency (sub-200ms TTS, millisecond-accurate transcription), quality (multiple tiers for accuracy and naturalness), and scale (multiplexing conversations over single WebSocket connections for efficient handling of high concurrency).