Keeping 20,000 GPUs healthy

This post details Modal's GPU reliability system, covering instance type testing and selection with observed performance and reliability differences across cloud providers (e.g., Cloud A's simple API, Cloud C's H100 overheating, Cloud D's A10 clock slowdowns and ECC errors). It discusses machine image preparation with automated testing (DCGM, custom GPU tests) and the trade-offs in instance boot checks (lightweight checks like `nvidia-smi` vs. deep `dcgmi diag`). The post elaborates on lifetime management through passive (dmesg, dcgmi health) and active healthchecking (dcgmi diag level 2, GPUBurn, NCCL tests), including actions taken upon failure (marking host unhealthy, draining, disposing/reinstalling). Observability is enhanced with a dashboard showing memory usage, utilization, temperature, and power usage metrics.