Heaps do lie: debugging a memory leak in vLLM. | Mistral AI

This post details the debugging of a memory leak in vLLM, specifically within a disaggregated Prefill/Decode serving setup utilizing NIXL and UCX. The investigation involved advanced profiling tools like Heaptrack and kernel-level tracing with pmap and BPFtrace to identify memory growth outside of the traditional heap, specifically in anonymous memory mappings managed by mmap/mremap. The root cause was identified as a leak within the KV Cache transfer mechanism via NIXL, leading to unreleased memory.