Modal's serverless Servers | Modal Blog

What this post added

Introduced Modal Servers, a new capability for running ultra-low-latency HTTP, WebSocket, and gRPC traffic. This feature provides a regionalized, autoscaling pool of server replicas behind Modal's routing layer, significantly reducing p50 latency from 39ms to 6ms compared to Web Functions. The architecture involves an L4 NLB, Envoy for TLS termination and HTTP/2 normalization, and a custom in-house proxy `fprs` built with Pingora for domain association and load balancing. This design prioritizes minimizing overhead and pushing queueing/retries to the application layer to achieve bleeding-edge latency for use cases like LLM inference.