Modal's serverless Servers | Modal Blog

Introduces Modal Servers, a new capability for running ultra-low-latency HTTP, WebSocket, and gRPC traffic. This system is designed to minimize overhead compared to Web Functions, reducing p50 latency from 39ms to 6ms. The architecture involves an L4 NLB, an Envoy edge proxy for TLS termination and HTTP/2 conversion, and a custom in-house proxy called `fprs` (built with Pingora) for domain association and load balancing to user containers. Key design principles include maximizing resource sharing while minimizing interference and avoiding network calls on the request path. The `fprs` proxy handles rapid churn in upstreams and integrates core features like auth, dynamic replica placement, regional routing, and autoscaling without putting control-plane lookups or queues in the hot path.