High performance ML inference with NVIDIA TensorRT

This post details the integration and benefits of NVIDIA TensorRT and TensorRT-LLM for achieving high-performance ML inference. It explains the role of TensorRT in the ML inference stack, how to use it for production serving, and provides performance benchmarks for popular open-source models (Mixtral 8x7B and SDXL) optimized with TensorRT. It emphasizes the importance of GPU hardware, batch size, precision, and input/output shapes for TensorRT optimization. The post also highlights Baseten's collaboration with NVIDIA and the productization of TensorRT support within Truss, their open-source model packaging framework, offering out-of-the-box features like in-flight batching for popular models.