Fine-tuning Gemma 2 with Keras - and an update from Hugging Face- Google Developers Blog

This post details the fine-tuning of Gemma 2 models using Keras and JAX, focusing on distributed fine-tuning with Model Parallelism and LoRA. It introduces the `keras.distribution.ModelParallel` API for specifying weight sharding configurations and demonstrates how to use `keras.distribution.LayoutMap` for layer-by-layer weight partitioning. It also showcases the use of LoRA for efficient fine-tuning by freezing model weights and replacing them with low-rank adapters. Furthermore, it announces enhanced Hugging Face integration allowing KerasNLP to load fine-tuned weights from Hugging Face for supported models, including Gemma and Llama3, and highlights the availability of PaliGemma for vision-language tasks.