LLM Serving Fairness: No More Noisy Neighbors | Cohere

This post details Cohere's implementation of a Serving Fairness system to manage inference requests fairly across tenants on a multi-tenant SaaS platform. It introduces a layered approach combining a Rate Limiter for admission control with three selectors: Performance Tier, Deficit Round Robin (DRR), and Priority. DRR is highlighted as the core mechanism for equitable distribution of compute within tiers, using either request-based or token-based budgeting to measure fairness. The post explains the trade-offs and use cases for each budgeting method and how priority queues maintain urgency within a tenant's fair share.