Autoscaling endpoints for LLM inference

This post elaborates on the autoscaling capabilities for LLM inference endpoints, detailing the trade-offs between over- and under-provisioning. It explains how to choose appropriate scaling metrics (concurrency-driven, SLO-driven, efficiency-driven) and tune scaling windows (`scale_up_window`, `scale_down_window`) to balance cost and latency. It also covers idle shutdown behavior, cold start phases, and the interaction of autoscaling with rollouts and traffic splitting.