Introducing Modal Auto Endpoints: Optimized inference you actually own | Modal Blog

Introduces Modal Auto Endpoints, a self-serve system for deploying production-grade LLM inference. This feature provides users with transparent access to the underlying inference code, metrics, and engine tuning parameters. It leverages Modal's existing AI infrastructure, including high-performance autoscaling and custom container runtime, and introduces Modal Servers for ultra-low-latency routing. The post details the use of advanced speculative decoding techniques like DFlash, highlights engine-level observability with metrics for server performance and inference specifics (TTFT, ITL, speculative decoding acceptance length), and discusses the ongoing development of an agentic system for automated inference endpoint optimization. It also mentions the ability to train custom speculators for specific production data.