On-device GenAI in Chrome, Chromebook Plus, and Pixel Watch with LiteRT-LM- Google Developers Blog

This post introduces LiteRT-LM, an open-source C++-based LLM pipeline framework that uses LiteRT for on-device inference. It enables running large language models (LLMs) like Gemini Nano and Gemma on edge devices, powering features in Chrome, Chromebook Plus, and Pixel Watch. Key technical contributions include: 1. The Engine/Session architecture for managing shared LLMs and feature-specific customization via LoRAs, enabling efficient context switching and session cloning with Copy-on-Write (CoW) KV-cache. 2. A modular design allowing for the creation of specialized, lean pipelines for resource-constrained devices by selecting minimum required components. 3. Cross-platform compatibility and hardware acceleration support (CPU, GPU, NPU) through the LiteRT runtime. The post provides access to the C++ preview interface, sample code, and documentation for developers to build custom LLM pipelines.