Faster Mixtral inference with TensorRT-LLM and quantization

This post details the optimization of Mixtral 8x7B inference by leveraging TensorRT-LLM and int8 quantization. It demonstrates achieving a time-to-first-token below 200ms and over 50 perceived tokens per second on a single A100 GPU. The comparison highlights a 2-3x increase in tokens per second and a 30% decrease in time-to-first-token when using TensorRT-LLM over vLLM for single requests. Quantization to int8 is shown to halve inference costs with a negligible impact on model quality (0.08% perplexity increase).