22,580: GPT-2 to Kimi K3, explained

What this post added

This post explains the architectural evolution from GPT-2 to Kimi K3, focusing on advancements in attention mechanisms. It details the inefficiency of standard decoder-only generation without caching, introduces the KV cache and its memory-bandwidth bottleneck, and then explains linear attention as a solution that replaces the growing KV cache with a fixed DxD state, reducing memory traffic for long sequences. It also introduces DeltaNet (Fast Weight Programmers) as a method to address information interference in linear attention when sequence length exceeds capacity, by learning to dynamically interact with and selectively update memory contents.