SUNK: Production-Grade AI Training at Scale

This post introduces SUNK (Slurm on Kubernetes), a new unified training system that integrates Slurm with Kubernetes to provide production-grade AI training at scale. Key contributions include topology-aware scheduling, automated health management and failure mitigation, deep operational visibility, and unified lifecycle management. It highlights improvements in goodput (up to 96%), Mean Time to Failure (10x longer for thousand-GPU clusters), and job restart times (~90 seconds). The post also mentions SUNK User Provisioning (SUP) for automated user onboarding and permissions, and integration with CoreWeave Mission Control for continuous health management and observability.