Using fractional H100 GPUs for efficient model serving

This post introduces the use of NVIDIA H100 GPUs with Multi-Instance GPU (MIG) technology for efficient model serving. It details how MIG allows splitting a single H100 GPU into two independent serving instances, each with approximately half the compute and memory of a full GPU. The post compares H100 MIG instances to A100 GPUs, highlighting advantages such as comparable or better performance for TensorRT-optimized models, support for FP8 precision, and increased flexibility. Benchmarks for Stable Diffusion XL and Mistral 7B demonstrate these performance benefits. It also outlines when H100 MIG is suitable (e.g., smaller models, FP8, TensorRT optimization) and when full GPUs are preferred (e.g., large models, large batch sizes, unoptimized implementations).