How Perplexity Brought Voice Search to Millions Using the Realtime API | OpenAI Developers

This post details Perplexity's production use of OpenAI's Realtime API (specifically Realtime-1.5) to power voice search for millions of users. Key technical contributions include: 1. Context Management Strategy: Shifting from large, all-or-nothing context updates to smaller, incremental chunks (2,000 tokens) to prevent catastrophic context loss. It also emphasizes the importance of correct role assignment (`system`, `user`, `assistant`) for context items to maintain natural interaction semantics. 2. Audio Standardization: Development of a Rust SDK to abstract client-specific audio differences, ensuring a consistent audio contract to the API by resampling to 48 kHz mono, matching Opus codec and WebRTC rates, and applying WebRTC APM for echo cancellation, noise reduction, and filtering before encoding. 3. Tuning for Messy Environments: Calibrating Voice Activity Detection (VAD) against real-world conditions (e.g., noisy bars) and introducing a 'voice lock' mechanism to manage pauses and user turn-taking in complex workflows. 4. Tool Usage Discipline: Narrowing toolset to core functions and providing explicit system prompt instructions for tool invocation, ensuring tool outputs are formatted as structured JSON rather than conversational dialogue for stability.