Routing LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAI

This post details the transition from a proportional controller for LLM inference routing weights to a control plane/data plane architecture. The new system uses an optimizer with a global view to minimize end-to-end latency, accounting for network distance and engine queueing. It introduces mechanisms like outlier penalties, retry budgets, and load shedding to manage traffic and prevent oscillations that disrupt KV cache locality.