Configuring Dedicated Model Inference

What this post added

Introduced the architecture for dedicated model inference, comprising endpoints, deployments, and configs. Detailed the capacity-aware traffic split mechanism, where traffic is routed proportionally to a deployment's effective capacity (weight * ready_replicas). Explained how this enables seamless autoscaling composition, controlled per-replica load, and robust handling of non-ready replicas. Clarified the distinctions between traffic splits, A/B experiment percents, and canary step percentages. Provided an experiment demonstrating capacity-aware routing and a comparison of latency vs. throughput profiles for different configurations.