TechPrufer
Get started
Hugging Face
Hugging Face logoHugging Face · Feature trail

Local AI Inference Stack

3 posts · Dec 2025 – May 2026

What this trail covers

Hugging Face is integrating GGML and llama.cpp to enhance the local AI inference stack. This initiative aims to streamline the process of shipping models from the Transformers library to llama.cpp for local execution, improve packaging and user experience for GGML-based software, and make local inference more accessible and ubiquitous for casual users. The long-term vision is to provide the community with building blocks for open-source superintelligence accessible globally. The llama.cpp server now supports GGUF quants, enabling Transformers to run these quantized models directly.

All Hugging Face trails
  1. Reachy Mini goes fully localThis post details how to run a full speech-to-speech pipeline locally for the Reachy Mini robot. It provides specific commands and configurations for using llama.cpp with Gemma 4 for LLM inference, Silero VAD, Parakeet-TDT for STT, and Qwen3-TTS. It also explains how to connect the Reachy Mini to this local backend. The post elaborates on the benefits of running a local pipeline (privacy, cost, control) and offers detailed instructions for using llama.cpp, vLLM, and Hugging Face Inference Endpoints as LLM backends for the speech-to-speech engine, emphasizing the importance of the Responses API protocol for decoupling and tool-call streaming.
  2. Transformers now runs llama.cpp quantsThis post announces that Hugging Face Transformers now supports running llama.cpp GGUF quantized models directly. This integration leverages the existing `local-ai-inference-stack` capability by enabling the use of GGUF-quantized models, which are typically smaller and faster for local inference, directly within the Transformers library. The post highlights the `unsloth/Qwen3.5-4B-GGUF` model as an example, demonstrating the ability to load and run these quantized models, thereby expanding the options for local AI inference.
  3. GGML and llama.cpp join HF to ensure the long-term progress of Local AIThis post announces the joining of GGML, creators of llama.cpp, to Hugging Face. The primary technical focus is on improving the seamless integration between Hugging Face's Transformers library and llama.cpp for local inference. This includes making it easier to ship new models from Transformers to llama.cpp, improving packaging and user experience for GGML-based software, and making local inference more accessible to casual users. The long-term vision is to provide building blocks for open-source superintelligence accessible globally.
  4. New in llama.cpp: Model ManagementIntroduced router mode to llama.cpp server, enabling dynamic model management (loading, unloading, switching) without server restarts. Implemented a multi-process architecture for model isolation. Added auto-discovery of GGUF models from cache or a specified directory (`--models-dir`). Implemented on-demand loading and LRU eviction (`--models-max`). Added API endpoints for listing (`/models`), loading (`/models/load`), and unloading (`/models/unload`) models. Introduced support for per-model settings via presets.ini.