H100 vs. H200 GPUs

This post details the technical comparison between NVIDIA H100 and H200 GPUs, focusing on their specifications relevant to AI inference. It explains how node-level VRAM (e.g., 640GB for 8x H100 SXM vs. 1128GB for 8x H200 SXM) dictates model fit, using GLM 5.2 (744B parameters) as an example requiring H200 for its weight size and KV cache headroom. The post also covers Multi-Instance GPU (MIG) for partitioning GPUs to serve smaller models cost-effectively and discusses async programming techniques (leveraging Tensor Memory Accelerators) to overlap data loading and computation, thereby improving GPU utilization and throughput. It provides guidance on choosing between H100 and H200 based on model size, traffic volume, and budget, highlighting H100 for smaller models/sporadic traffic and H200 for very large models and long-context inference.