Compressing KV cache memory by half with sparse attention

This post introduces techniques to compress KV cache memory by half in LLMs using sparse attention. It details the application of synthetic long instruction data generation for fine-tuning, selective dense layers based on sensitivity analysis, specialized sliding-window attention with global sink tokens, and auxiliary memory tokens. The authors release two models (Llama-3-CBHybridL-8B and Llama-3-CBHybridM-8B) and the synthetic dataset used for fine-tuning. The contribution focuses on the methods for selecting sensitive layers for dense attention and the use of memory tokens to enhance long-context performance.