One-second voice-to-voice latency with Modal, Pipecat, and open models

This post demonstrates building a low-latency voice AI chatbot using Pipecat and open models, leveraging Modal's ability to autoscale independent GPU and CPU services for STT, LLM, and TTS inference, and integrating with WebSockets for real-time communication. It details strategies for achieving sub-second voice-to-voice latency by optimizing STT (Parakeet-tdt-v3), LLM inference (Qwen3-4B-Instruct-2507 with vLLM), and TTS (KokoroTTS), and coordinating these with the Pipecat framework over WebSockets. The architecture highlights the separation of concerns between the stateful bot container (CPU) and the compute-intensive AI services (GPU), enabling cost-effective scaling.