Speech-to-Text (STT) APIs

5/29/2026

This post details the technical optimizations implemented to achieve industry-leading speech-to-text performance. Key contributions include: 1. TensorRT multi-profile engines for the encoder to handle variable audio shapes efficiently. 2. Conditional CUDA graphs to move the decoder's conditional branching onto the GPU, eliminating CPU round trips. 3. Collapsing CPU preprocessing into fewer processes and using custom protocols over Unix domain sockets or shared memory for lower-copy data paths. 4. Implementing evented I/O with `epoll` for streaming to reduce scheduler pressure and improve predictability. 5. Utilizing `gc.freeze()` to prevent Python's garbage collector from causing latency spikes in streaming workflows.

5/14/2026

This post introduces Violin, an open-source video translation tool built on Together AI's infrastructure. It details the pipeline's three core stages: ASR using Together's Whisper V3 endpoint, LLM translation leveraging Deepseek V4 Pro with user-defined translation rules, and TTS using Cartesia's Sonic 3 for natural-sounding voice generation. It also describes the integration of a video chat assistant powered by a vision-language model (Qwen3.5-397B-A17B) that understands both audio and visual content. The post highlights the availability of Violin as a web app, CLI tool, and agent skill, emphasizing its open-source nature.

4/2/2026

This post announces the native deployment of Deepgram's STT and TTS models (Nova-3, Nova-3 Multilingual, Flux, Aura-2) on Together AI's Dedicated Model Inference. It highlights the technical benefits of Flux for conversational STT with 250ms end-of-turn detection, Nova-3 for production audio transcription with vocabulary customization, Nova-3 Multilingual for cross-language support, and Aura-2 for clear and consistent TTS. The integration allows for a unified voice pipeline on a single production platform, reducing latency and operational fragility by co-locating STT, LLM, and TTS workloads.

3/12/2026

This post details the integration of Cartesia Sonic-3/Sonic-2 and Deepgram's STT (Nova-3, Nova-3 Multilingual, Flux) and TTS (Aura-2) models into the Together AI platform. It emphasizes the architectural benefits of co-locating STT, LLM, and TTS components within the same cluster to achieve end-to-end latency under 500ms for real-time voice agents. The post highlights the modular design, flexibility in model choice, and production-ready features like zero data retention, SOC 2 Type II, and HIPAA compliance.

2/23/2026

This post details research into the failure modes of Automatic Speech Recognition (ASR) systems, specifically their inability to reliably transcribe street names for speakers with diverse linguistic backgrounds. It introduces new benchmarks (SF Streets and US Streets) and quantifies the performance gap, showing an average 39% transcription error rate for street names across 15 state-of-the-art models, with an 18% accuracy gap for non-English primary speakers. The post proposes and demonstrates a synthetic data generation technique using cross-lingual style transfer to improve accuracy by up to 60% with fewer than 1,000 training samples, highlighting a practical method for enhancing ASR robustness for critical applications.

11/4/2025

This post introduces significant enhancements to Together AI's voice capabilities, focusing on real-time voice agents. Key contributions include: 1. Streaming Whisper STT: Optimized inference and system design (WebSocket streaming, VAD tuning) for up to 35% faster complete transcripts compared to alternatives, prioritizing time-to-complete-transcript for voice agents. 2. Serverless Open-Source TTS: Introduction of Orpheus (187ms TTFB, natural voice) and Kokoro (97ms TTFB, ultra-low latency) available via REST and WebSocket APIs, eliminating the need for dedicated infrastructure for sporadic TTS needs. 3. Voxtral Mini and Speaker Diarization: Addition of a premium multilingual transcription model optimized for European languages and challenging audio, offering measurably lower word error rates than standard Whisper, and speaker diarization for identifying and labeling different speakers in audio. Architectural decisions emphasize low latency (sub-200ms TTS, millisecond-accurate transcription), quality (accuracy and natural-sounding speech), and scale (consistent performance under load via optimized WebSocket multiplexing and auto-scaling).

7/10/2025

This post introduces Together AI's new Speech-to-Text (STT) APIs, focusing on the technical optimizations that enable 15x faster transcription than OpenAI's offering. Key technical details include the use of Silero for smart voice activity detection, intelligent chunking and batching strategies for long audio files, and engine improvements to maximize GPU utilization for the Whisper V3 Large model. The post also highlights production-ready API features like enterprise-scale file handling (over 1GB), superior word-level alignment, comprehensive language support (50+ languages), dedicated endpoints for low latency, and batch processing capabilities. It positions STT as a foundational element for voice AI applications, integrating with existing LLM services.