TechPrufer
Sign inGet started

Blogs›Modal›WebSockets

WebSockets

WebSockets

3
posts
2024–2026

Modal functions now support the WebSocket protocol, enabling real-time, bidirectional data transfer between clients and servers. This feature allows for persistent connections, reducing latency and facilitating real-time updates for applications. It integrates seamlessly with existing Modal functions, supporting libraries like FastAPI and automatically scaling based on input volume. Use cases include real-time streaming responses for audio and text generation, status updates on long-running task. Modal now also offers ultra-low-latency Servers for HTTP, WebSocket, and gRPC traffic, providing a regionalized, autoscaling pool of server replicas behind Modal's routing layer. This new Server capability reduces p50 latency from 39ms to 6ms by minimizing overhead and pushing queueing/retries to the application layer, making it ideal for applications like LLM inference where every millisecond counts. The Server architecture involves an L4 NLB, an Envoy edge proxy for TLS termination and HTTP/2 normalization, and an in-house proxy `fprs` built with Pingora for domain association and load balancing across compute plane replicas.

2026

Modal's serverless Servers | Modal Blog

6/25/2026

Introduced Modal Servers, a new capability for running ultra-low-latency HTTP, WebSocket, and gRPC traffic. This feature provides a regionalized, autoscaling pool of server replicas behind Modal's routing layer, significantly reducing p50 latency from 39ms to 6ms compared to Web Functions. The architecture involves an L4 NLB, Envoy for TLS termination and HTTP/2 normalization, and a custom in-house proxy `fprs` built with Pingora for domain association and load balancing. This design prioritizes minimizing overhead and pushing queueing/retries to the application layer to achieve bleeding-edge latency for use cases like LLM inference.

2024

Introducing: WebSockets on Modal

2/27/2024

Introduces native support for WebSocket connections within Modal functions. Demonstrates setup using FastAPI and highlights the automatic scaling benefits for real-time applications. Discusses use cases such as real-time streaming responses, long-running task status updates, and hosting interactive frameworks.

Product updates: WebSocket support, interactive commands & more

2/15/2024

This post announces the general availability of WebSocket support for Modal functions, enabling real-time, bidirectional data transfer. It highlights the ability to host frameworks like Streamlit and provides a link to documentation for further details.