Gemma explained: What’s new in Gemma 2- Google Developers Blog

This post introduces Gemma 2, a new suite of open AI models available in 2B, 9B, and 27B parameter sizes. It details architectural innovations including alternating local and global attention, logit soft-capping, RMSNorm for normalization, and Grouped-Query Attention (GQA). The post highlights performance advantages of knowledge distillation and GQA over MHA, and the benefits of deeper models. It also provides code examples for Gemma 27B and notes on GQA head configurations for different model sizes.