Gemma explained: What’s new in Gemma 2- Google Developers Blog

What this post added

This post introduces Gemma 2, detailing its architectural innovations including alternating local and global attention, logit soft-capping, RMSNorm for pre and post-normalization, and Grouped-Query Attention (GQA). It provides specific implementation details for the Gemma 27B model's self-attention mechanism and MLP layers, including the number of heads and head sizes for GQA across different model sizes (27B, 9B, 2B). Key findings from the technical report are summarized, highlighting the benefits of knowledge distillation, GQA over MHA, and deeper vs. wider model architectures.