TechPrufer
Get started
Hugging Face
Hugging Face logoHugging Face · Feature trail

Mixture of Experts (MoE) Models

4 posts · Feb 2025 – Oct 2026

What this trail covers

Hugging Face now supports Mixture of Experts (MoE) models, enabling more efficient and scalable large language models by selectively activating expert sub-networks for different inputs. This capability allows for models with a vast number of parameters that can be trained and served with significantly reduced computational cost compared to dense models of equivalent size. The DeepSeek-R1 model is highlighted as an example of a large MoE model available on the Hub. This post introduces Olmo-core and Falcon-Emirati-7B, a dialect-specialized model built on Falcon-H1-Arabic, demonstrating the application of advanced architectures for nuanced language understanding and generation. The Falcon-H1 architecture combines State Space Models (Mamba) and Transformer attention for efficient long-sequence processing. The development of Falcon-Emirati-7B involved a dedicated data pipeline with authentic dialectal web data, MSA data about Emirati culture, and synthetically generated data guided by glossaries and style rules. Evaluation included manual assessment by native speakers and automatic evaluation on the Alyah benchmark, with LLM-as-judge evaluation for pairwise comparisons. The 7B parameter size was chosen as a balance between quality and cost for dialect specialization.

All Hugging Face trails
  1. Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEsThis post introduces Olmo-core 3, a new open-source training infrastructure specifically designed for large Mixture-of-Experts (MoE) models. It details the architectural shift from Fully Sharded Data Parallelism (FSDP) to Distributed Data Parallelism (DDP) for MoE training, which keeps experts resident on GPUs and routes data to them, improving throughput. Key optimizations include row-wise expert parallelism, GPU-resident routing, and grouped GEMM. The post also highlights the benefits of MXFP8 for reducing computation and memory usage, showing a 21% throughput increase and reduced peak memory compared to BF16. Benchmarks demonstrate scaling to trillion-parameter models with high TFLOP/s/GPU. The post also discusses observed failure modes like 'token gerrymandering' and the importance of matching input values for performance comparisons.
  2. Falcon-Emirati: When an LLM Learns the Dialect, the Culture, and the NuanceThis post details the development of Falcon-Emirati-7B, a dialect-specialized LLM built on the Falcon-H1-Arabic model. It highlights the challenges of dialect adaptation due to the spoken nature of Emirati Arabic and the reliance on cultural context. The data pipeline involved curating authentic dialectal web data, MSA data on Emirati culture, and synthetically generated data with strict rules. The 7B parameter size was selected for its balance of quality and cost. Evaluation methodologies included manual and LLM-as-judge assessments. The underlying Falcon-H1 architecture, which fuses Mamba and Transformer attention, is also discussed as a foundation for handling morphologically rich languages and long contexts.
  3. Mixture of Experts (MoEs) in TransformersThis post introduces asynchronous routing and load balancing strategies for Mixture of Experts (MoE) models. It details how to manage the dynamic selection and distribution of expert computations across available hardware, aiming to optimize inference throughput and latency. The DeepSeek-R1 model is used as a concrete example to illustrate these techniques.
  4. Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCLThis post details the implementation of asynchronous GRPO (Proximal Policy Optimization) with LoRA (Low-Rank Adaptation) for training Mixture of Experts (MoE) models across Hugging Face Jobs. It highlights a novel approach that bypasses traditional NCCL for distributed communication, instead utilizing a bucket and proxy mechanism. This enables efficient distributed training of MoE models without requiring direct GPU-to-GPU communication via NCCL, potentially reducing infrastructure complexity and cost.