Model autoscaling features on Baseten

This post introduces and details Baseten's model autoscaling features. It explains how users can configure minimum and maximum replicas to control costs and handle traffic bursts, with excess traffic being queued. The concept of 'scale to zero' is explained, where models can be put to sleep after inactivity. Specific autoscaling settings are detailed: autoscaling window (timeframe for analyzing traffic), scale-down delay (additional time before scaling down), and concurrency target (requests per replica). The post also highlights Baseten's fast cold start times, citing 9 seconds for Stable Diffusion XL on an A100, and provides load test graphs demonstrating the autoscaling features in action.