TechPrufer
Sign inGet started

Blogs›Cohere›LLM Serving Fairness

LLM Serving Fairness

LLM Serving Fairness

1
posts
2026

Cohere has implemented a "Serving Fairness" system to manage inference requests fairly across tenants on a multi-tenant SaaS platform. This system addresses the "noisy neighbor" problem where one customer's traffic spikes can negatively impact others. It employs a layered approach combining admission control (Rate Limiter) with a series of selectors (Performance Tier, Deficit Round Robin, and Priority) to ensure equitable distribution of GPU resources. The Deficit Round Robin algorithm, with configurable request-based or token-based budgeting, is central to ensuring fair capacity allocation within tiers. Priority and deadline ordering are maintained within each tenant's fair share.

2026

LLM Serving Fairness: No More Noisy Neighbors | Cohere

6/17/2026

This post introduces Cohere's "Serving Fairness" solution for LLM inference. It details a four-mechanism system: a Rate Limiter for admission control, Performance Tier for SLA-based prioritization, Deficit Round Robin (DRR) for equitable resource distribution within tiers, and a Priority selector for ordering requests within a tenant's allocated share. The post elaborates on DRR's implementation with request-based and token-based budgeting, explaining their suitability for different endpoint types (generative vs. batched). It highlights the integration of these familiar mechanisms to achieve burst-proof, fair capacity sharing without sacrificing tenant-specific urgency.