Agentic Inference: Prefix Caching and Routing | CoreWeave

This post introduces and details the concepts of prefix caching and prefix-aware routing for agentic inference. It explains how agentic prompts consist of a shared prefix and new content, and how reusing the KV cache for the prefix can significantly reduce prefill computation. The post emphasizes that prefix-aware routing is crucial to ensure that inference requests are directed to workers holding the relevant cached KV state, thereby avoiding cache misses. It outlines four key infrastructure considerations for production scale: routing to cached prefixes, preserving cache locality during scaling, balancing cache locality with load, and implementing robust observability for metrics like cache hit rate and TTFT. The post positions these techniques as essential for building infrastructure specifically designed for agentic workloads.