TechPrufer
Get started
OpenAI
OpenAI logoOpenAI · Feature trail

AI-Powered Voice and Audio Capabilities

3 posts · Sep 2025 – Mar 2026

What this trail covers

This post continues the evolution of OpenAI's AI-powered voice and audio capabilities, encompassing speech-to-text, text-to-speech, and real-time speech-to-speech interactions. The Realtime API and its associated speech-to-speech model have graduated to General Availability (GA) with significant improvements in model quality, reliability, and developer ergonomics. Key updates include API shape changes with a new GA interface, removal of the `temperature` parameter from the GA interface due to its unique behavior with this model architecture, and the introduction of several new features. These new features include conversation idle timeouts for better turn detection, extended session lengths (up to 60 minutes) and larger token windows (32,768 tokens) for the `gpt-realtime` model, configurable truncation settings for managing long conversations and their impact on prompt caching, asynchronous function calling with placeholder responses for improved UX, EU data residency support, enhanced tracing capabilities in the developer console, hosted prompts for easier prompt management and versioning, and sideband connection options for secure server-side logic integration via WebRTC or SIP.

All OpenAI trails
  1. How Perplexity Brought Voice Search to Millions Using the Realtime API | OpenAI DevelopersThis post details Perplexity's production use of OpenAI's Realtime API (specifically Realtime-1.5) to power voice search for millions of users. Key technical contributions include: 1. Context Management Strategy: Shifting from large, all-or-nothing context updates to smaller, incremental chunks (2,000 tokens) to prevent catastrophic context loss. It also emphasizes the importance of correct role assignment (`system`, `user`, `assistant`) for context items to maintain natural interaction semantics. 2. Audio Standardization: Development of a Rust SDK to abstract client-specific audio differences, ensuring a consistent audio contract to the API by resampling to 48 kHz mono, matching Opus codec and WebRTC rates, and applying WebRTC APM for echo cancellation, noise reduction, and filtering before encoding. 3. Tuning for Messy Environments: Calibrating Voice Activity Detection (VAD) against real-world conditions (e.g., noisy bars) and introducing a 'voice lock' mechanism to manage pauses and user turn-taking in complex workflows. 4. Tool Usage Discipline: Narrowing toolset to core functions and providing explicit system prompt instructions for tool invocation, ensuring tool outputs are formatted as structured JSON rather than conversational dialogue for stability.
  2. Updates for developers building with voice | OpenAI DevelopersThis post introduces new audio model snapshots (`gpt-4o-mini-transcribe-2025-12-15`, `gpt-4o-mini-tts-2025-12-15`, `gpt-realtime-mini-2025-12-15`, `gpt-audio-mini-2025-12-15`) that significantly improve reliability and quality for production voice workflows. Key technical advancements include lower word-error rates for transcription, fewer hallucinations with background noise, more natural and stable voice output for TTS, and improved instruction-following and tool-calling accuracy for real-time speech-to-speech models. These updates address common failure modes in voice applications, such as long conversations and edge cases like silence, and offer cost optimization opportunities by enabling the use of mini models for certain applications.
  3. Developer notes on the Realtime API | OpenAI DevelopersThis post details the General Availability (GA) launch of the Realtime API and the `gpt-realtime` speech-to-speech model. It highlights API shape changes, including a migration path from beta to GA, and the removal of the `temperature` parameter from the GA interface, with guidance to use prompting for controlling response characteristics. New features introduced include conversation idle timeouts (`idle_timeout_ms`), extended session lengths (60 mins) and token windows (32,768 tokens) with configurable truncation strategies (e.g., `retention_ratio`) to manage context and prompt caching, asynchronous function calling with placeholder responses, EU data residency support, new trace event types (session updated, output text generation), hosted prompts for managing prompts by ID and version, and sideband connection options for WebRTC/SIP to secure business logic on the server.