GPU Died. Training Didn't: Self-Healing Training at Scale — Crusoe

This post introduces a novel architecture that combines Slurm and Kubernetes to create a self-healing training infrastructure. It details the challenges of traditional Slurm in dynamic environments and how Kubernetes addresses these gaps through node health checks, observability, and automatic remediation. The core contribution is the automatic remediation flow: notify, drain, replace, requeue, and resume, demonstrated by recovering from an XID 79 GPU failure and resuming training in under 15 minutes without human action. It also touches on sharing GPUs between training and inference.