Mistral 7B

This post announces the release of Mistral 7B, a 7.3B parameter language model. It details its performance advantages over Llama 2 13B and Llama 1 34B across various benchmarks (reasoning, knowledge, comprehension, math, code). It introduces and explains the technical implementation of Grouped-query attention (GQA) for faster inference and Sliding Window Attention (SWA) for handling longer sequences with linear compute cost and reduced cache memory. It also presents a fine-tuned version, Mistral 7B Instruct, demonstrating its generalization capabilities.