Benchmarking fast Mistral 7B inference

This post details Baseten's best-in-class performance for Mistral 7B inference, achieving a 130ms TTFT, 170 TPS, and 700ms total response time for 100 tokens. It elaborates on benchmarking methodologies, including the impact of batch sizes and sequence lengths on latency and throughput, and emphasizes the advantages of dedicated model deployments. The post also discusses techniques like session reuse for reducing TTFT and the importance of accurate tokenization for TPS measurement.