Large Language Models On-Device with MediaPipe and TensorFlow Lite- Google Developers Blog

Introduces the experimental MediaPipe LLM Inference API for running Large Language Models (LLMs) fully on-device across web, Android, and iOS. Details the steps for integration, including model conversion to TensorFlow Lite Flatbuffer and SDK usage. Highlights supported LLMs (Gemma, Phi 2, Falcon, Stable LM) and provides performance benchmarks (Time to First Token, Decode Speed). Discusses performance optimizations such as weight sharing, optimized fully connected ops (I8MM instructions), balancing compute and memory, custom operators for GPU acceleration, pseudo-dynamism in attention blocks, and optimizations for specific models like Gemma 2B (int4 quantization).