AI-Powered Voice and Audio Capabilities

3/25/2026

This post details Perplexity's implementation of OpenAI's Realtime API for voice search, focusing on technical challenges and solutions. Key contributions include strategies for context management with large transcripts by breaking them into smaller chunks and carefully managing message roles (`system`, `user`, `assistant`); standardizing audio processing across different client stacks using a Rust SDK for resampling, codec preference, and noise reduction; tuning voice activity detection (VAD) for noisy environments and implementing a 'voice lock' mechanism to manage pauses; and optimizing tool usage by narrowing the toolset and formatting tool outputs as structured JSON. The post emphasizes the importance of these technical decisions in bringing a robust voice interface to millions of users.

12/22/2025

Introduced new model snapshots for speech-to-text, text-to-speech, and real-time speech-to-speech, focusing on improving reliability and quality for production voice workflows. Key improvements include lower word-error rates in noisy audio, fewer hallucinations during silence or background noise, and more natural and stable voice output. Specifically, the `gpt-realtime-mini` model shows significant gains in instruction-following accuracy (18.6 pp) and tool-calling accuracy (12.9 pp) for real-time agents. Text-to-speech models achieve roughly 35% lower WER on benchmarks like Common Voice and FLEURS. Speech-to-text models produce ~90% fewer hallucinations compared to Whisper v2 in hallucination-with-noise evaluations. Custom Voices also benefit from more natural tones and increased faithfulness.

9/12/2025

This post details the General Availability (GA) launch of the Realtime API and the `gpt-realtime` speech-to-speech model. It highlights API shape changes, including a migration from beta to GA interfaces, and details new features such as image input, long context support (up to 60 minutes session duration, 32,768 token window), asynchronous function calling with placeholder responses, EU data residency options, and hosted prompts. It also introduces conversation idle timeouts with server-side VAD, configurable truncation settings for long conversations (including a cache-friendly retention ratio), and improved tracing events. The post also discusses changes to temperature parameter behavior and the introduction of sideband connections for WebRTC and SIP.