Model Routing Is Simple. Until It Isn’t.

This post details the challenges and solutions for building an effective model routing system for AI agents, shifting from a classification problem to a systems optimization problem. It highlights that cost is more than model pricing due to caching, complexity is more than task difficulty due to hidden execution steps and governance, and latency is more than model speed due to routing overhead and granularity. The authors present an optimization-based approach that balances cost, quality, and latency, demonstrating its effectiveness on the AppWorld Test Challenge with a CodeAct agent.