Validate GPU Cluster Readiness Before AI Workloads Land | NVIDIA Technical Blog

This post introduces the NVIDIA Cluster Readiness Engine (NVCRE), an open-source Kubernetes controller designed to validate GPU cluster readiness for AI workloads before production deployment. NVCRE utilizes a layered API (Certification, Workflow, Job) to run real distributed workloads across topology-aware node groups, measuring performance against defined pass criteria (e.g., NCCL bandwidth, training goodput ratio). It features adaptive fault isolation to pinpoint failing nodes and a WorkloadRun API for simplifying the setup of multi-node GPU workloads on Kubernetes. NVCRE integrates with NVIDIA AI Cluster Runtime and NVSentinel for comprehensive validation and monitoring.