Large Language Models On-Device with MediaPipe and TensorFlow Lite- Google Developers Blog

What this post added

Introduces the experimental MediaPipe LLM Inference API, enabling Large Language Models (LLMs) to run fully on-device across Web, Android, and iOS platforms. Details the process of converting model weights to TensorFlow Lite Flatbuffers using the MediaPipe Python Package and integrating the LLM Inference SDK. Highlights supported LLMs (Gemma, Phi 2, Falcon, Stable LM) and their parameter sizes. Presents performance benchmarks for Time to First Token and Decode Speed, with detailed explanations of optimizations such as weight sharing, optimized Fully Connected Ops (int8/int4 quantization, ARM v9 I8MM instructions), balancing compute and memory for prefill/decode phases, custom GPU operators, pseudo-dynamism, and optimized activation data types.