22,580: GPT-2 to Kimi K3, explained

This post details the architectural evolution from GPT-2 to Kimi K3, highlighting key innovations that enabled massive scale. It explains the fundamental components of GPT-2's transformer architecture, including token and positional embeddings, transformer blocks, and causal self-attention. The post introduces the KV cache as a solution to redundant computation in autoregressive decoding and discusses its memory-bandwidth bottleneck. It then delves into linear attention as an alternative to softmax attention, explaining its reassociable property and the trade-off between expressiveness and efficiency. Finally, it introduces DeltaNet (fast weight programmers) as a mechanism to address information interference in additive caches when sequence length exceeds storage capacity, further optimizing for large-scale models.