How KV caches impact time to first token for LLMs

This post details the impact of KV (key-value) caching on LLM Time To First Token (TTFT) latency and throughput. It explains how KV caching, by storing intermediate attention computations, reduces the complexity of subsequent token generation from quadratic to linear. The post presents experimental results showing that reusing KV caches across LLM calls can save approximately 0.15ms per cached input token, leading to significant TTFT improvements (e.g., 100ms for 1000 cached tokens). This optimization is crucial for delivering a faster and more responsive Glean Assistant experience.