Two Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story — Asaf Gardin & Yuval Belfer

This post details the debugging of two critical bugs in vLLM that caused model outputs to degrade into gibberish under specific load and memory conditions. The first bug was identified as a scheduler issue where decode operations were executed before prefill operations, impacting Mamba's stateful computations. The second bug was a 32-bit integer overflow in the Mamba state cache index, which wrapped around after four billion operations. Both issues were surfaced by memory pressure and diagnosed using logprob comparisons and request ID tracing. The fixes involved ensuring correct execution order and using a 64-bit integer type (size_t) for the index.