Mixtral Sparse Mixture of Experts Model

3/16/2026

Mistral Small 4 introduces a unified model architecture combining reasoning, multimodal, and agentic coding capabilities. It utilizes a Mixture of Experts (MoE) design with 128 experts, 4 active per token, and has 119B total parameters with 6B active. The model supports a 256k context window and features a configurable `reasoning_effort` parameter. Performance improvements include a 40% reduction in latency and a 3x increase in throughput compared to its predecessor. Native multimodality is integrated, accepting both text and image inputs. The post details hardware requirements for enterprise deployment and highlights optimizations for vLLM and SGLang through collaboration with NVIDIA.

12/2/2025

This post announces Mistral 3, a new generation of models. It details Mistral Large 3, a sparse mixture-of-experts model, and the Ministral 3 series. It highlights the collaboration with NVIDIA, vLLM, and Red Hat for optimized training and inference, including specific technologies like llm-compressor, TensorRT-LLM, SGLang, and support for disaggregated serving and speculative decoding. The post also mentions the multimodal and multilingual capabilities of the new models and their availability on various platforms.

5/7/2025

Introduces Mistral Medium 3, a new model class focused on balancing performance and cost. Details its performance benchmarks against competitors like Claude Sonnet 3.7, Llama 4 Maverick, and Cohere Command A, highlighting its strengths in coding and STEM. Mentions enterprise capabilities such as hybrid/on-premises deployment and custom post-training. Discusses availability on Mistral La Plateforme and Amazon Sagemaker.

3/17/2025

Mistral Small 3.1 is introduced, featuring improved text performance, multimodal understanding, and an expanded context window of up to 128k tokens. It demonstrates performance exceeding comparable models like Gemma 3 and GPT-4o Mini, with inference speeds of 150 tokens per second. The post details instruct and pretrained performance across text, multimodal, multilingual, and long context benchmarks. It highlights use cases such as conversational assistance, image understanding, and function calling, emphasizing its suitability for on-device deployment and fine-tuning for specialized domains. The release includes both base and instruct checkpoints.

9/17/2024

Introduces Mistral Small v24.09, an upgraded enterprise-grade small model with 22 billion parameters, offering improved human alignment, reasoning, and code capabilities. The model is released under the Mistral Research License and supports self-deployment for non-commercial use cases.

7/24/2024

Introduces Mistral Large 2, a 123B parameter model with a 128k context window. Highlights its performance on MMLU (84.0% accuracy), code generation benchmarks (on par with GPT-4o, Claude 3 Opus, Llama 3 405B), mathematical reasoning (GSM8K, MATH), and general alignment benchmarks (MT-Bench, Wild Bench, Arena Hard). Details improvements in instruction following, multilingual capabilities (especially on multilingual MMLU), and function calling/retrieval. Notes the model's single-node inference capability and cost efficiency. Mentions availability on la Plateforme (`mistral-large-2407`) and via cloud providers, and the extension of fine-tuning capabilities.

7/18/2024

Introduces Mistral NeMo, a 12B parameter model with a 128k context window, highlighting its performance on reasoning, world knowledge, and coding benchmarks. Details the new Tekken tokenizer, which offers improved compression rates for various languages and source code. Presents instruction fine-tuning results demonstrating enhanced capabilities in instruction following, reasoning, and code generation. Mentions availability through HuggingFace, la Plateforme, and NVIDIA NIM.

4/17/2024

This post announces and details Mixtral 8x22B, a new open-source SMoE model. It highlights its parameter efficiency (39B active out of 141B), multilingual capabilities, function calling support, and large context window (64K tokens). Performance benchmarks are provided for reasoning, knowledge, multilingual tasks, maths, and coding, comparing it against other open models. The model is released under Apache 2.0.

12/11/2023

Introduces Mixtral 8x7B, a sparse mixture of experts (SMoE) model. Details the architecture where a router network selects two out of eight experts per token at each layer, enabling a large total parameter count (46.7B) with a significantly smaller active parameter count per token (12.9B) for efficient inference. Discusses pre-training on open web data and presents benchmark results comparing Mixtral to Llama 2 and GPT3.5, highlighting performance, multilingual capabilities, and bias metrics. Also announces an instruction-tuned version and integration with vLLM for open-source deployment.

9/27/2023

This post announces the release of Mistral 7B, a 7.3B parameter model that outperforms open models up to 13B parameters on English and code benchmarks. It highlights the development of a new MLops stack and a sophisticated data processing pipeline from scratch. The post also reiterates Mistral AI's commitment to open models and mentions ongoing work on larger models and novel architectures.

9/27/2023

Introduces Mistral 7B, a 7.3B parameter model, detailing its performance advantages over Llama 2 13B and Llama 1 34B. Highlights the use of Grouped-query attention (GQA) for faster inference and Sliding Window Attention (SWA) for efficient handling of longer sequences. Explains the technical details of SWA, including its linear compute cost and how stacked layers can attend beyond the window size. Discusses the optimization of attention cache using rotating buffers. Presents Mistral 7B Instruct, a fine-tuned version for chat, and its performance on MT-Bench.