TechPrufer
Get started
DataRobot
DataRobot logoDataRobot · Feature trail

TokenGrid: LLM Token Scheduling and Capacity Management

1 post · Jul 2026 – Aug 2026

What this trail covers

TokenGrid is a stateful, token-aware capacity scheduler for managing LLM inference costs and resource utilization. It operates in the admission path, bridging gateways and compute resources. Features include dynamic rate limiting based on multi-dimensional quotas (tokens per second, monthly token budgets, container memory), a fair-sharing engine that classifies workloads, enforces tenant quotas, and applies memory-based throttling. It also supports various service classes (dedicated, guaranteed, elastic, spot, and preemptible). This post details its application in multi-tenant inference environments, addressing issues of stranded GPU capacity, undifferentiated rate limiting, and production latency spikes during traffic bursts. It introduces debt-based fairness for temporarily throttled workloads and capacity expressed in inference-native units like token throughput, KV cache, and concurrency, running as a Kubernetes-native layer above vLLM or TensorRT-LLM.

All DataRobot trails
  1. Stop Rate-Limiting Requests. Start Scheduling Tokens: Introducing DataRobot TokenGridThis post introduces DataRobot TokenGrid, a stateful, token-aware capacity scheduler designed to address the limitations of traditional rate-limiting systems in LLM inference. It replaces static hardware partitions with a shared, work-conserving capacity pool. Key contributions include: 1. Dynamic rate limiting based on multi-dimensional quotas (tokens per second, monthly token budgets, container memory). 2. A fair-sharing engine that classifies workloads by estimating prefill demand, enforcing tenant quotas, and applying memory-based backpressure. 3. Automated capacity reconciliation by feeding container performance signals into a centralized Token Admin Service to recalibrate execution slots and TPM/RPM caps. This system aims to optimize GPU infrastructure utilization and govern costs across self-hosted and third-party LLM APIs.
  2. A decade of open source at DataRobot: from predictive AI to the agent lifecycleThis post introduces Token Pool, an open-source project that addresses multi-tenant inference challenges at the API gateway. It expresses capacity in inference-native units (token throughput, KV cache, concurrency) and implements a debt-based fairness mechanism to prevent tenant starvation and monopolization during overload. It details how Token Pool can hold sub-1.2 second P99 time-to-first-token for guaranteed workloads by throttling spot traffic, contrasting with baseline degradation. The system runs as a Kubernetes-native layer above vLLM or TensorRT-LLM.