LLM Serving Fairness: No More Noisy Neighbors | Cohere

What this post added

This post introduces Cohere's "Serving Fairness" solution for LLM inference. It details a four-mechanism system: a Rate Limiter for admission control, Performance Tier for SLA-based prioritization, Deficit Round Robin (DRR) for equitable resource distribution within tiers, and a Priority selector for ordering requests within a tenant's allocated share. The post elaborates on DRR's implementation with request-based and token-based budgeting, explaining their suitability for different endpoint types (generative vs. batched). It highlights the integration of these familiar mechanisms to achieve burst-proof, fair capacity sharing without sacrificing tenant-specific urgency.