AI Edge Torch Generative API for Custom LLMs on Device- Google Developers Blog

What this post added

Introduced the AI Edge Torch Generative API, enabling developers to author high-performance LLMs in PyTorch for deployment on edge devices using the TensorFlow Lite (TFLite) runtime. This API offers custom transformer support, optimized CPU performance, compatibility with TFLite deployment flows, and support for models like TinyLlama, Phi-2, and Gemma 2B. It features multi-signature export for prefill and decode operations to optimize serving performance and includes LLM-specific performance optimizations such as high-performance SDPA and KVCache, leveraging TFLite's XNNPack delegate, and reducing runtime memory consumption.