TechPrufer
Get started
Modal
Modal logoModal · Feature trail

WebSockets

3 posts · Feb 2024 – Jun 2026

What this trail covers

Modal functions now support the WebSocket protocol, enabling real-time, bidirectional data transfer between clients and servers. This feature allows for persistent connections, reducing latency and facilitating real-time updates for applications. It integrates seamlessly with existing Modal functions, supporting libraries like FastAPI and automatically scaling based on input volume. Use cases include real-time streaming responses for audio and text generation, status updates on long-running task. This post demonstrates building a low-latency voice AI chatbot using Pipecat and open models, leveraging Modal's ability to autoscale independent GPU and CPU services for STT, LLM, and TTS inference, and integrating with WebSockets for real-time communication.

All Modal trails
  1. Modal's serverless Servers | Modal BlogIntroduces Modal Servers, a new capability for running ultra-low-latency HTTP, WebSocket, and gRPC traffic. This system is designed to minimize overhead compared to Web Functions, reducing p50 latency from 39ms to 6ms. The architecture involves an L4 NLB, an Envoy edge proxy for TLS termination and HTTP/2 conversion, and a custom in-house proxy called `fprs` (built with Pingora) for domain association and load balancing to user containers. Key design principles include maximizing resource sharing while minimizing interference and avoiding network calls on the request path. The `fprs` proxy handles rapid churn in upstreams and integrates core features like auth, dynamic replica placement, regional routing, and autoscaling without putting control-plane lookups or queues in the hot path.
  2. Real-time inference for robots at Physical Intelligence | Modal BlogPhysical Intelligence has implemented a specialized transport for real-time robotic inference, moving beyond standard Modal Tunnels (TCP) to a QUIC-based portal running over UDP with automatic NAT traversal. This system establishes a persistent, bidirectional channel between the robot runtime and the Modal GPU container, streaming observations and receiving action outputs over the same channel. It utilizes STUN discovery and UDP hole punching coordinated via Modal Dict for rendezvous, enabling direct connections even behind NAT. Communication is over QUIC, implemented in Rust, resulting in approximately 10-15 ms of network overhead for cloud inference. This approach avoids TCP's potential for request-response stalls in the control loop, which is critical for robotic applications.
  3. One-second voice-to-voice latency with Modal, Pipecat, and open modelsThis post demonstrates building a low-latency voice AI chatbot using Pipecat and open models, leveraging Modal's ability to autoscale independent GPU and CPU services for STT, LLM, and TTS inference, and integrating with WebSockets for real-time communication. It details strategies for achieving sub-second voice-to-voice latency by optimizing STT (Parakeet-tdt-v3), LLM inference (Qwen3-4B-Instruct-2507 with vLLM), and TTS (KokoroTTS), and coordinating these with the Pipecat framework over WebSockets. The architecture highlights the separation of concerns between the stateful bot container (CPU) and the compute-intensive AI services (GPU), enabling cost-effective scaling.
  4. Lambda on hard mode: Inside Modal's web infrastructureThis post details the `modal-http` service, which acts as the ingress for web requests (HTTP and WebSockets) to Modal functions. It explains the challenges of serving compute-heavy, long-running serverless workloads via web requests, contrasting Modal's approach with traditional serverless platforms. The post describes the system's design as a distributed operating system, translating web requests into function calls rather than relying solely on HTTP reverse proxies. It highlights the use of Rust for performance and complexity management in handling HTTP edge cases and large inputs, and discusses the system's ability to support complex heterogeneous workloads on CPU and GPU.
  5. Introducing: WebSockets on ModalIntroduces WebSocket support for Modal functions, allowing for real-time, bidirectional data transfer. Provides a code example using FastAPI and explains how Modal's automatic scaling applies to WebSocket handlers. Highlights use cases such as real-time streaming responses, status updates on long-running tasks, and hosting open-source frameworks like ComfyUI, Streamlit, and Gradio.
  7. Product updates: WebSocket support, interactive commands & moreAdds WebSocket protocol support to Modal functions, enabling real-time, bidirectional data transfer and persistent connections. This facilitates real-time updates and integrates with frameworks like Streamlit.