Running fine-tuned models on Workers AI with LoRAs

This post introduces the open beta of Bring Your Own (BYO) LoRAs on Workers AI, enabling users to run fine-tuned inference with LoRA adapters for Mistral, Gemma, and Llama 2 models. It explains the concepts of fine-tuning and LoRA, detailing the mathematical principles behind LoRA's efficiency. The post outlines how to use LoRAs with Workers AI, including supported base models and current limitations (e.g., no quantized models, adapter size limits). It also delves into the technical implementation of multi-tenant LoRA serving, highlighting the use of the Punica CUDA kernel design and global cache optimizations to achieve low latency and high throughput.