TechPrufer
Get started
Together AI
Together AI logoTogether AI · Feature trail

Cache-Aware Prefill-Decode Disaggregation (CPD)

1 post · Mar 2026

What this trail covers

Together AI has developed Cache-Aware Prefill-Decode Disaggregation (CPD), a serving architecture that separates cold and warm workloads based on cache hit rate. CPD utilizes a three-level KV-cache hierarchy (GPU memory, host DRAM, distributed cache) and a dedicated pre-prefill tier for cold requests, while normal prefill nodes handle warm requests by fetching KV blocks from the cache. This architecture aims to improve sustainable throughput and reduce time-to-first-token (TTFT) for long-context inference, delivering up to 40% higher sustainable throughput under mixed, real-world traffic.

All Together AI trails
  1. Cache-aware prefill–decode disaggregation (CPD) for up to 40% faster long-context LLM servingThis post introduces Cache-Aware Prefill-Decode Disaggregation (CPD), an extension to standard prefill-decode disaggregation. CPD introduces a dedicated pre-prefill tier for cold requests that compute new context and write KV cache to a distributed cache, and normal prefill nodes that prioritize warm requests by reading KV blocks from the cache. It relies on a three-level KV-cache hierarchy (GPU memory, host DRAM, distributed cache) and a CPD-aware router to distinguish warm and cold requests. Evaluations show CPD improves sustainable QPS by up to 35-40% over existing disaggregated designs and maintains tighter tail latency bounds under mixed warm and cold workloads.