Gemma explained: What’s new in Gemma 3- Google Developers Blog

This post details the architectural changes in Gemma 3, focusing on its new vision-language capabilities, including the use of a SigLIP encoder and a "Pan&Scan" algorithm for image processing. It explains how images are represented as "soft tokens" using a MultiModalProjector to reduce inference resources. The post also describes architectural modifications for memory efficiency, such as 5-to-1 interleaved attention (5 local attention layers with sliding window and 1 global attention layer) and the adoption of QK-norm instead of soft-capping for improved accuracy and speed. The extended context length support (32k to 128k tokens) is highlighted as a result of these changes. The post contrasts Gemma 3's attention mechanism with PaliGemma's, and introduces a new tokenizer with a larger vocabulary (262k) for enhanced multilingual support, noting its similarity to Gemini's tokenizer. It also provides a detailed code snippet of the Gemma 3 27B model architecture.