TechPrufer
Get started
Cohere
Cohere logoCohere · Feature trail

LLM Serving Fairness

2 posts · Jun 2026

What this trail covers

Cohere's Serving Fairness system manages inference requests fairly across tenants on a multi-tenant SaaS platform, addressing the 'noisy neighbor' problem. It employs a layered approach combining admission control (Rate Limiter) with a series of selectors (Performance Tier, Deficit Round Robin, and Priority) to ensure equitable distribution of GPU resources. The Deficit Round Robin algorithm, with request-based or token-based budgeting, ensures fair sharing of compute resources within tiers, while priority queues handle urgency within a tenant's allocated share.

All Cohere trails
  1. LLM Serving Fairness: No More Noisy Neighbors | CohereThis post details Cohere's implementation of a Serving Fairness system to manage inference requests fairly across tenants on a multi-tenant SaaS platform. It introduces a layered approach combining a Rate Limiter for admission control with three selectors: Performance Tier, Deficit Round Robin (DRR), and Priority. DRR is highlighted as the core mechanism for equitable distribution of compute within tiers, using either request-based or token-based budgeting to measure fairness. The post explains the trade-offs and use cases for each budgeting method and how priority queues maintain urgency within a tenant's fair share.