LLM Inference Routing Policy

This post introduces a new capability focused on routing LLM inference traffic in production. It moves from a feedback-loop-based proportional controller to a control plane with a global view and a data plane for serving cached routing weights. The optimizer aims to minimize end-to-end latency under hard constraints, considering factors beyond simple network proximity. Protections like outlier penalties, retry budgets, and load shedding are also detailed.