Distributed Training Resiliency

7/30/2026

This post introduces new capabilities for CoreWeave SUNK, a Slurm-on-Kubernetes system for AI research clusters. Key additions include SUNK self-service for faster cluster deployment and standardized configurations, and SUNK Anywhere for extending the unified training system to external infrastructure. It also details enhancements to automated user provisioning (AUP and SUP) for simplified and secure access management, and the integration of GPU straggler detection into Mission Control observability for deeper visibility into distributed training performance bottlenecks.

7/30/2026

This post details the common failure modes in large-scale AI training: stragglers, synchronization stalls, and stalled GPUs. It explains how these issues manifest as quiet failures (allocated but not utilized compute) and discusses their root causes, such as degraded hardware, network variability, and data pipeline bottlenecks. The post emphasizes the importance of moving beyond generic cluster health monitoring to per-rank metrics, communication vs. compute time, and I/O latency. It advocates for an architectural approach to prevent these coordination failures, highlighting CoreWeave Cloud's integrated infrastructure, orchestration, and observability as key to reducing variance and improving training reliability and efficiency.

7/30/2026

This post introduces a reference architecture for distributed AI training that emphasizes a purpose-built, layered approach over general-purpose infrastructure. It details four key interdependent layers: topology-aware orchestration for optimized job placement and fault containment, checkpoint-optimized storage for frequent and high-throughput writes to minimize recovery loss, high-performance interconnects for efficient GPU communication and collective offload, and integrated observability for cross-layer visibility and proactive fault detection. The post argues that this integrated system design is crucial for achieving reliability and performance at production scale, contrasting it with fragmented, bolted-on solutions.

7/30/2026

This post elaborates on the challenges and solutions for achieving high ROI in distributed AI training. It identifies key bottlenecks including execution consistency, utilization vs. throughput, and networking/storage limitations. The post argues that purpose-built infrastructure, as exemplified by CoreWeave's architecture, is crucial for converting allocated capacity into measurable model progress. It references external evaluations like SemiAnalysis ClusterMAX and MLPerf to support the claim that CoreWeave's infrastructure delivers sustained effective throughput and performance at scale, directly impacting training ROI.

7/30/2026

This post defines and elaborates on the challenges and failure modes of distributed training at scale. It identifies three primary failure categories: compute, coordination, and recovery. It also highlights the diagnostic challenge of NCCL errors masking underlying issues. The post uses data from Epoch AI and Meta FAIR to illustrate the scaling of training times, model parameters, and the inverse relationship between cluster size and MTTF. It also references a ByteDance case study on straggler impact. The post advocates for a system-level approach to building resiliency for large-scale distributed training.

5/20/2026

This post provides practical guidance on improving AI infrastructure resilience by detailing four key strategies: 1) designing training code to tolerate node failures through checkpointing and containerization, leveraging orchestration systems like Slurm for automatic resubmission; 2) implementing comprehensive instrumentation across hardware and software layers to differentiate infrastructure and application errors, using tools like Grafana and Weights & Biases to correlate metrics like MFU with system events; 3) automating alerts and recovery mechanisms, citing internal testing that shows automated recovery can be 3x faster than manual intervention, and suggesting heartbeat monitors, hang detectors, and retry thresholds; and 4) stress-testing infrastructure with NCCL tests, burn-in/out tests, and controlled failure simulations to identify friction points and validate recovery behavior before production workloads.

5/20/2026

CoreWeave joins Red Hat's llm-d OSS project as a founding member, contributing to the advancement of distributed AI inference on Kubernetes. This initiative aims to enhance vLLM capabilities with features like Prefill and Decode Disaggregation, lmcache-based KV cache offloading, an AI Inference Gateway, and AI-aware network routing using NIXL. CoreWeave will contribute Kubernetes native operators, testing and benchmarking harnesses, and an enhanced Tensorizer for faster model loading, building upon their expertise in large-scale Kubernetes and AI inference deployments to improve LLM serving, reduce latency, optimize costs, and scale AI workloads across different cloud platforms.

5/20/2026

This post details a six-week benchmarking study on NVIDIA H100 GPUs at CoreWeave, focusing on achieving high Model FLOPs Utilization (MFU) and Mean Time To Failure (MTTF) for large-scale AI training. It highlights specific infrastructure optimizations including a dual-fabric network (InfiniBand + DPU-offloaded Ethernet), topology-aware scheduling with SUNK, automated re-queuing of failed jobs, and Tensorizer-based asynchronous checkpointing which reduced save times from 129s to 17s. The study achieved 51-52% MFU (vs. 35-45% typical) and 3.66 days MTTF at 1,024 GPUs (10x improvement), with projected improvements at 16,384 GPUs. It also details custom tokenizer performance and third-party validation against published results.

5/20/2026

This post provides concrete benchmark results from a large-scale pretraining run (30-billion-parameter LLM across 1,024 GPUs) validating the effectiveness of CoreWeave's purpose-built AI infrastructure. It quantifies performance improvements with metrics like 51-52% MFU, 97-98% ETTR, and a 10x improved MTTF (3.66 days vs. 0.33 days industry average). These results demonstrate a 7-15 day faster time-to-market for a 30-day training cycle, reinforcing the benefits of vertical integration, dual network fabrics, topology-aware orchestration, and high-speed data pipelines for AI training.

3/4/2026

This post introduces SUNK (Slurm on Kubernetes), a unified training system that integrates Slurm with Kubernetes to provide cloud-native agility, operational ease, and scalable lifecycle management for AI training. It details SUNK's capabilities including topology-aware scheduling, automated health management and failure mitigation, deep operational visibility, and unified lifecycle management. The post highlights SUNK's proven performance at scale, citing metrics like 96% goodput and 10x longer MTTF for large clusters, and its integration with CoreWeave Mission Control for continuous health management and automated remediation. It also mentions guided self-service for deploying and managing SUNK via the CoreWeave console using a Kubernetes operator.