LiteRT: The Universal Framework for On-Device AI- Google Developers Blog

This post announces the full production release of LiteRT as the universal on-device inference framework for the AI era, building upon its TensorFlow Lite foundation. Key advancements include 1.4x faster GPU performance than TFLite, new NPU acceleration, a unified workflow for GPU and NPU acceleration across Android, iOS, macOS, Windows, Linux, and Web, superior cross-platform GenAI deployment for models like Gemma, and first-class PyTorch/JAX support via seamless model conversion. It highlights comprehensive GPU support via OpenCL, OpenGL, Metal, and WebGPU through ML Drift, achieving average 1.4x faster performance than TFLite GPU delegate, with optimizations like asynchronous execution and zero-copy buffer interoperability yielding up to 2x faster performance. LiteRT simplifies NPU integration with a three-step process and offers AOT and JIT compilation. Integrations with MediaTek and Qualcomm NPUs achieve significant speed improvements. The post details LiteRT's support for popular open-weight models and its integrated tech stack for GenAI deployment, including LiteRT Torch Generative API, LiteRT-LM, and LiteRT Converter & Runtime.