Multi-Rack NVIDIA Vera Rubin NVL72 Bring-Up | CoreWeave

This post details the engineering challenges and solutions involved in bringing up multi-rack NVIDIA Vera Rubin NVL72 clusters. It highlights the automation of hardware lifecycle management from initial detection to production readiness, including firmware updates and configuration. The post emphasizes the importance of rack-level performance validation, identifying and troubleshooting outliers before they become stragglers in distributed workloads. It also describes the construction of a non-blocking RoCE network fabric across racks, utilizing ConnectX-9 SuperNICs and a two-tier, multi-rail, multi-plane architecture to achieve 1.6 Tb/s of backend network connectivity per GPU and scale to approximately 128,000 GPUs.