Unlocking the full power of NVIDIA H100 GPUs for ML inference with TensorRT

This post details Baseten's benchmarking of NVIDIA H100 GPUs for ML inference using TensorRT and TensorRT-LLM. It quantifies performance improvements over A100 GPUs for LLMs (Mistral 7B) and image generation models (Stable Diffusion XL), highlighting gains in throughput and reductions in latency. The post explains how H100 architecture features and TensorRT optimizations contribute to these gains, and discusses cost savings achievable by migrating A100 workloads to H100s. Baseten now offers optimized model inference on H100 GPUs.