Streamlining LLM Inference at the Edge with TFLite- Google Developers Blog

Introduced an improved XNNPack weight cache provider for TensorFlow Lite. This provider utilizes memory-mapped files (mmap) to persist packed weights to disk, enabling them to be loaded across multiple TFLite interpreter instances and even across processes. This eliminates the overhead of repacking weights on each model load, significantly reducing startup latency and peak memory usage. It also allows for weight deduplication for models with shared subgraphs and simplifies the user-facing API by requiring only a cache file path.