Serving DeepSeek-V4: why million-token context is an inference systems problem

What this post added

This post details the serving implications of DeepSeek-V4's million-token context window, focusing on its hybrid attention design (CSA, HCA, SWA). It explains how V4 compresses the token axis of the KV cache to reduce memory pressure and improve concurrency. The post highlights the need for inference engines to manage multiple KV-cache layouts, adapt prefix caching policies, and consider regime-dependent performance (long-context decode vs. short-context prefill). It also emphasizes that the same model weights require different serving profiles based on workload characteristics (long-context agents, coding agents, short chat, RL rollouts). The key technical contributions are the analysis of V4's architectural impact on KV cache management and the identification of critical benchmarking areas for long-context models.