AI Edge Torch Generative API for Custom LLMs on Device- Google Developers Blog

Introduces the AI Edge Torch Generative API, enabling developers to author high-performance LLMs in PyTorch for deployment using the TensorFlow Lite (TFLite) runtime. This API offers custom transformer support, optimized CPU performance (with GPU/NPU support coming), compatibility with TFLite deployment flows (quantization, runtime), and support for models like TinyLlama, Phi-2, and Gemma 2B. It provides an easy authoring experience with core building blocks, weight re-mapping, intuitive quantization APIs, and multi-signature export for disaggregated prefill/decode functions. Performance optimizations include high-performant SDPA and KVCache, leveraging TFLite's XNNPack delegate, avoiding wasteful computations, and reducing runtime memory consumption. The post details the conversion and quantization process to TFLite, including export to StableHLO and further compiler passes.