TechPrufer
Get started
Mistral AI
Mistral AI logoMistral AI · Feature trail

Mixtral Sparse Mixture of Experts Model

12 posts · Sep 2023 – Oct 2026

What this trail covers

Mistral AI continues to evolve its model offerings with the Mistral 4 family, including a 1 trillion-parameter natively multimodal model with 49 billion active parameters. This model, ML4, demonstrates exceptional performance across coding, agentic workflows, and multimodal understanding, competitive with the strongest open-source models globally and outperforming any open-weight model developed in the US or Europe. It is trained on Mistral's own datacenters in Europe and will be available with open weights. ML4 shows state-of-the-art performance in critical enterprise workloads like cybersecurity, finance, and law, and surpasses frontier closed models in domains like visual grounding. It is also proficient in scientific tasks, math, and knowledge work, with third-party evaluations showing it exceeds GPT-6-Astra in legal and financial benchmarks.

All Mistral AI trails
  1. Introducing Mistral Large 4 | MistralThis post announces Mistral Large 4 (ML4), a 1 trillion-parameter natively multimodal model with 49 billion active parameters. It highlights ML4's frontier performance in coding (DeepSWE, Terminal Bench 4.0, SWE-Atlas-QnA), agentic workflows (AutomationBench, AA-Briefcase), multimodal understanding (Dense 200, ChartQA Pro, GDP.pdf), science and math (SciCode-Verified, STEM tasks), and knowledge work (Finance Agent v2, Finch, Harvey's Legal Agent Benchmark). The model was trained on 3,800 NVIDIA Grace Blackwell GPUs in Mistral's European datacenters and will have its weights released by the end of the month. It is currently available in public preview via API on Mistral Studio.
  2. Introducing Mistral Small 4 | Mistral AIIntroduces Mistral Small 4, a unified model combining reasoning, multimodal, and agentic coding capabilities. Key architectural details include a Mixture of Experts (MoE) with 128 experts (4 active per token), 119B total parameters (6B active), and a 256k context window. It features configurable reasoning effort via the `reasoning_effort` parameter and native multimodality. Performance highlights include a 40% reduction in end-to-end completion time and 3x more requests per second compared to Mistral Small 3. The post details enterprise-grade efficiency requirements (NVIDIA HGX H100/H200, DGX B200) and emphasizes its open-source nature with availability on various inference engines and optimization with NVIDIA NIM and NeMo.
  3. Introducing Mistral 3 | Mistral AIThis post announces the release of Mistral 3, a new generation of models including dense models (Ministral 3: 3B, 8B, 14B) and Mistral Large 3, a sparse mixture-of-experts model with 41B active and 675B total parameters. All models are released under the Apache 2.0 license. Mistral Large 3 is highlighted as a top-tier open-weight model with image understanding and strong multilingual capabilities, achieving #2 on the LMArena leaderboard for OSS non-reasoning models. The release details significant collaboration with NVIDIA for hardware-software co-design, enabling efficient inference with TensorRT-LLM and SGLang, and optimized serving for MoE architectures on Blackwell NVL72 systems and single-node configurations using vLLM. Ministral models are optimized for edge deployments on various NVIDIA hardware. The post also mentions a reasoning version of Mistral Large 3 is forthcoming.
  4. Mistral Small 3.1 | Mistral AIThis post announces Mistral Small 3.1, an iteration on the Mistral Small 3 model. Key improvements include enhanced text performance, multimodal understanding capabilities, and an expanded context window up to 128k tokens. The model claims to outperform comparable models like Gemma 3 and GPT-4o Mini, achieving inference speeds of 150 tokens per second. It is released under an Apache 2.0 license and is suitable for on-device use cases, conversational assistance, function calling, and fine-tuning. The post also mentions the availability of base and instruct checkpoints for further customization.
  5. Mistral Small 3 | Mistral AIThis post introduces Mistral Small 3, a new 24B-parameter model released under Apache 2.0. It is designed for low latency and efficient local deployment, achieving over 81% accuracy on MMLU and 150 tokens/s latency. The model is competitive with larger models like Llama 3.3 70B and Qwen 32B, and serves as an open replacement for proprietary models like GPT4o-mini. It is released as both a pre-trained and instruction-tuned checkpoint, intended as a base for further reasoning capabilities. The post details performance benchmarks, including human evaluations and instruct performance across various benchmarks, and outlines use cases such as fast-response conversational assistance, low-latency function calling, fine-tuning for subject matter expertise, and local inference. It is available on la Plateforme and through partners like Hugging Face, Ollama, and Together AI.
  7. Large Enough | Mistral AIIntroduces Mistral Large 2, a 123B parameter model with a 128k context window. Highlights its performance on MMLU (84.0% accuracy), code generation benchmarks (on par with GPT-4o, Claude 3 Opus, Llama 3 405B), mathematical reasoning (GSM8K, MATH), and general alignment benchmarks (MT-Bench, Wild Bench, Arena Hard). Details improvements in instruction following, multilingual capabilities (especially on multilingual MMLU), and function calling/retrieval. Notes the model's single-node inference capability and cost efficiency. Mentions availability on la Plateforme (`mistral-large-2407`) and via cloud providers, and the extension of fine-tuning capabilities.
  8. Mistral NeMo | Mistral AIThis post introduces Mistral NeMo, a 12B parameter model developed in collaboration with NVIDIA. Key technical contributions include a large context window of 128k tokens, state-of-the-art performance in reasoning, world knowledge, and coding accuracy for its size category, and FP8 inference enabled by quantization awareness. A new tokenizer, Tekken, is introduced, which is more efficient at compressing natural language text and source code across many languages compared to previous tokenizers. The model is released under the Apache 2.0 license as pre-trained base and instruction-tuned checkpoints, and is available via la Plateforme and as an NVIDIA NIM inference microservice.
  9. MathΣtral | Mistral AIIntroduces MathΣtral, a specialized 7B parameter model derived from Mistral 7B, focused on advanced mathematical and STEM reasoning. Achieves state-of-the-art performance in its size category on benchmarks like MATH (56.6%) and MMLU (63.47%). Demonstrates improved reasoning with inference-time computation, scoring 68.37% on MATH with majority voting and 74.59% with a strong reward model. Highlights the performance/speed tradeoffs of purpose-built models and provides weights for use and fine-tuning.
  10. Cheaper, Better, Faster, Stronger | Mistral AIThis post introduces Mixtral 8x22B, a new open-source sparse Mixture-of-Experts (SMoE) model. It highlights the model's cost efficiency (39B active parameters out of 141B total), multilingual fluency (English, French, Italian, German, Spanish), strong mathematics and coding capabilities, native function calling, and a 64K token context window. The post emphasizes its performance-to-cost ratio, claiming it's faster than dense 70B models and more capable than other open-weight models. It includes benchmark comparisons (MMLU, HellaSwag, Wino Grande, Arc Challenge, TriviaQA, NaturalQS, HumanEval, MBPP, GSM8K, Math) demonstrating its superiority in reasoning, knowledge, multilingual tasks, math, and coding against other leading open models like LLaMA 2 70B.
  11. Mixtral of experts | Mistral AIThis post introduces Mixtral 8x7B, a sparse mixture of experts (SMoE) model with open weights. It details the SMoE architecture where a router network selects two experts per layer for each token, enabling a large number of parameters (46.7B total) with a reduced active parameter count (12.9B per token) for efficient inference. Performance benchmarks against Llama 2 and GPT3.5 are provided, along with analysis of bias and multilingual capabilities. An instruction-following variant, Mixtral 8x7B Instruct, is also released. The post also mentions integration with vLLM for efficient inference and deployment via Skypilot.
  12. Mistral 7BThis post announces the release of Mistral 7B, a 7.3B parameter language model. It details its performance advantages over Llama 2 13B and Llama 1 34B across various benchmarks (reasoning, knowledge, comprehension, math, code). It introduces and explains the technical implementation of Grouped-query attention (GQA) for faster inference and Sliding Window Attention (SWA) for handling longer sequences with linear compute cost and reduced cache memory. It also presents a fine-tuned version, Mistral 7B Instruct, demonstrating its generalization capabilities.