Serving DeepSeek-V4: why million-token context is an inference systems problem

This post details the inference systems challenges and solutions for serving DeepSeek-V4, a model with a 1M-token context window. It focuses on the architectural shift to a serving-systems problem due to V4's Compressed Sparse Attention (CSA) / Heavily Compressed Attention (HCA) / Sliding Window Attention (SWA) design. Key contributions include: analysis of KV cache compression on the token axis to reduce memory pressure and improve concurrency; the requirement for multiple KV-cache layouts (CSA, HCA, SWA) and the associated memory management challenges; prefix caching becoming a policy decision across different cache objects; the regime-dependent performance of V4 (long-context decode vs. short-context prefill); the need for different serving profiles for the same weights based on workload shape (long-context agents, coding agents, short chat, RL rollouts); and recommendations for benchmarking V4 based on context-length regime, prefix reuse, cache policy, and endpoint profile.