2x faster inference with KV cache-aware routing

This post details the integration and performance benefits of NVIDIA Dynamo's KV cache-aware routing within Baseten's AI model serving platform. It explains how KV cache-aware routing works by directing incoming requests to model replicas with relevant cached context, minimizing recomputation. Benchmarks with Qwen3 Coder 480B demonstrate a 50% reduction in TTFT and a 34% reduction in TPOT, with P95/P99 latency decreasing by 48%/49% respectively. Throughput metrics show a 61% increase in RPS and a 62% increase in TPS. The post also mentions the flexibility of Dynamo to combine KV routing with other strategies and highlights ongoing work on KV cache offloading.