Build realtime voice agents on AI Gateway

What this post added

This post introduces and details the integration of audio/voice capabilities into Vercel's AI Gateway and AI SDK. It covers the implementation of real-time voice agents, text-to-speech, and speech-to-text functionalities. Technical details include the use of the `useRealtime` hook in the browser for managing WebSocket connections, microphone capture, and audio playback, along with server-side token generation for authentication. It also explains the underlying mechanics of real-time sessions, including turn detection and tool integration. For text-to-speech, it demonstrates the `generateSpeech` function for creating audio files. For speech-to-text, it showcases the `transcribe` function for converting audio to text. The post highlights how these new audio modalities leverage existing AI Gateway features like provider routing, observability, and spend controls.