Blazing fast on-device GenAI with LiteRT-LM- Google Developers Blog

This post introduces LiteRT-LM, a new runtime for on-device GenAI, specifically optimized for Gemma 4 models. It details performance improvements achieved through advanced quantization, XNNPACK/MLDrift kernels, and optimized data transfer avoidance. Key contributions include the implementation of Multi-Token Prediction (MTP) for a 2.2x speedup by executing drafter and primary models on the same hardware IP, advanced session management for KV cache serialization, and memory footprint optimizations for multimodal capabilities. It also highlights Thinking Mode for reasoning and constrained decoding for structured output, along with native function-calling support. New Swift and JavaScript APIs are introduced for iOS and web (WebGPU) respectively.