T5Gemma: A new collection of encoder-decoder Gemma models- Google Developers Blog

What this post added

Introduces T5Gemma, a new collection of encoder-decoder LLMs derived from decoder-only Gemma models via an adaptation technique. Details the model adaptation process, which involves initializing encoder-decoder parameters with weights from pretrained decoder-only models and further pre-training. Highlights T5Gemma's performance advantages in quality-inference efficiency across benchmarks like SuperGLUE and GSM8K, demonstrating comparable or better results than decoder-only Gemma models. Discusses the flexibility of 'unbalanced' encoder-decoder configurations (e.g., 9B encoder with 2B decoder) for task-specific trade-offs. Releases T5Gemma checkpoints in various sizes (Small, Base, Large, XL, 2B, 9B) and variants (pretrained, instruction-tuned), including an unbalanced 9B-2B model, and models trained with PrefixLM or UL2 objectives.