Evaluating NVIDIA H200 Tensor Core GPUs for LLM inference

This post provides a detailed benchmark analysis of NVIDIA H200 Tensor Core GPUs for LLM inference, comparing their performance against H100 GPUs. It highlights that H200s offer significant advantages for large models (100+ billion parameters), large batch sizes, and long input sequences, demonstrating up to 3.4x higher performance in specific long-context scenarios and 47% higher performance in high-throughput batch inference. For shorter contexts and outputs, performance gains are minimal. The analysis includes methodology using TensorRT-LLM and trtllm-bench, and discusses trade-offs between H100 and H200 for cost-efficiency in inference workloads.