Gemma explained: RecurrentGemma architecture- Google Developers Blog

This post details the RecurrentGemma architecture, a hybrid model that mixes gated linear recurrences with local sliding window attention, based on the Griffin architecture. It explains the trade-offs between transformer and recurrent models for long context prompts, the architectural components (embedding layer, decoder layers with recurrent and MQA blocks), and key parameters like model width, RNN width, MLP expansion factor, and local attention window size. It also provides a Python code snippet for the RecurrentGemmaForCausalLM model.