Cache-aware prefill–decode disaggregation (CPD) for up to 40% faster long-context LLM serving

What this post added

Introduced Cache-Aware Prefill-Decode Disaggregation (CPD) to improve long-context LLM serving. CPD separates cold and warm workloads by cache hit rate, utilizing a three-level KV-cache hierarchy (GPU memory, host DRAM, distributed cache via RDMA) and a router that steers requests to dedicated pre-prefill, prefill, or decode nodes. This architecture achieves up to 40% higher sustainable throughput and lower TTFT by preventing long-running cold prefills from blocking cache-reusing warm requests.