Gemma explained: An overview of Gemma model family architectures- Google Developers Blog

What this post added

This post introduces the Gemma family of models, detailing their architectures and design choices. It explains the decoder-only transformer architecture, core parameters like d_model, layers, feedforward hidden dimensions, num_heads, num_kv_heads (distinguishing MHA in 7B from MQA in 2B), head_size, and vocab_size. It provides a detailed breakdown of the Gemma 7B model's architecture, including its embedding layer, decoder layers, self-attention mechanism (with specific details on projections, heads, and rotary embeddings), MLP with GeGLU activation, and language modeling head. It also highlights the use of SentencePiece for tokenization and the context length of 8192 tokens.