Testing Llama 3.3 70B inference performance on NVIDIA GH200 in Lambda Cloud

This post details the performance testing of Llama 3.3 70B inference on NVIDIA GH200 hardware, highlighting the architectural advantages of the GH200's high-bandwidth interconnect between CPU and GPU. The experiment demonstrates that offloading KV cache to abundant CPU memory, enabled by the 450 GB/s NVLink-C2C interconnect, allows for KV cache re-use and prefill optimizations. Benchmarking against an H100 GPU using SGLang's tool and a ShareGPT dataset showed a 32% performance gain on the GH200, attributed to the larger effective KV cache capacity. The post also discusses the trade-offs between GH200 and H100/H200 configurations for serving large models.