Gemma explained: RecurrentGemma architecture- Google Developers Blog

What this post added

This post introduces the RecurrentGemma architecture, detailing its hybrid approach combining gated linear recurrences with local sliding window attention. It explains the architectural components like RecurrentGemmaDecoderLayer, RecurrentGemmaRecurrentBlock, RecurrentGemmaSdpaAttention, and RecurrentGemmaMlp. Key parameters such as non-embedding and embedding parameters, model width, RNN width, MLP expansion factor, and local attention window size are discussed. The post also provides a Python code snippet for the RecurrentGemmaForCausalLM model and elaborates on the embed_tokens layer and the structure of the 26 decoder layers, including the specific implementation of residual blocks with recurrent blocks and local MQA blocks.