Mixtral of experts | Mistral AI

What this post added

Introduces Mixtral 8x7B, a sparse mixture of experts (SMoE) model. Details the architecture where a router network selects two out of eight experts per token at each layer, enabling a large total parameter count (46.7B) with a significantly smaller active parameter count per token (12.9B) for efficient inference. Discusses pre-training on open web data and presents benchmark results comparing Mixtral to Llama 2 and GPT3.5, highlighting performance, multilingual capabilities, and bias metrics. Also announces an instruction-tuned version and integration with vLLM for open-source deployment.