TechPrufer
Get started
Coreweave
Coreweave logoCoreweave · Feature trail

Distributed Training Resiliency

23 posts · Mar 2026 – Oct 2026

What this trail covers

CoreWeave enhances its AI infrastructure for large-scale model training and inference by focusing on optimizing NVIDIA GPU clusters for both performance and reliability. This involves a multi-layered approach including bare-metal hardware, a dual-fabric network architecture (InfiniBand for compute, Ethernet for storage), topology-aware scheduling with automated node eviction and job rescheduling via SUNK (Slurm on Kubernetes), and optimized asynchronous checkpointing using Tensorizer. The SUNK system is further optimized with ART-Optimized Megatron (AOM), a new backend for the ART training library that achieves 12X higher training step throughput than previous backends by implementing shared prefix training, maximizing packed sequence length, and tuning parallelism configurations. This shared prefix approach, implemented using PyTorch's FlexAttention and custom GDN layer modifications, significantly increases trajectory density and reduces compute, memory, and memory bandwidth requirements per rollout, bringing performance closer to theoretical bounds.

All Coreweave trails
  1. ART-Optimized Megatron (AOM): 12x higher training throughputIntroduced ART-Optimized Megatron (AOM) as a new backend for the ART training library, achieving 12x higher training step throughput. Implemented shared prefix training using PyTorch's FlexAttention and custom Gated DeltaNet (GDN) layer modifications to handle non-linear tree layouts, significantly increasing trajectory density. Optimized packed sequence length for maximum GPU throughput and tuned parallelism configurations for improved trajectories per second. This builds upon existing distributed training resiliency efforts by directly enhancing the training throughput component.
  2. CoreWeave: Only Provider with 3x Platinum ClusterMAX | BlogThis post details CoreWeave's operational excellence and proactive failure detection mechanisms that contribute to their consistent Platinum ClusterMAX ratings. It highlights their proprietary stack for straggler detection and automated remediation, which goes beyond conventional health checks to identify and address silent degradations in GPU performance. The post emphasizes how this capability, combined with their integrated approach to compute, networking, storage, and orchestration, ensures maximized GPU utilization and accelerated time to production for AI workloads.
  3. Video Captioning at Scale: 600 TB in 95 Minutes | CoreWeave BlogThis post details the successful application of CoreWeave's distributed training resiliency infrastructure, specifically the integration of Anyscale's Ray Data with CoreWeave Kubernetes Service (CKS) and CoreWeave AI Object Storage (CAIOS) with LOTA, for a large-scale video captioning workload. It demonstrates processing 600 TB of data in 95 minutes using 1,600 GPUs. Key technical contributions include: 1. The rapid setup of a large-scale GPU cluster (1,600 NVIDIA RTX PRO 6000 Blackwell GPUs) and Anyscale on CKS in under 24 hours, leveraging CKS's managed control plane and pre-installed drivers/networking. 2. The transition from Ray Core to Ray Data for improved data ingestion and autoscaling, resulting in 3.7x more captions per GPU-hour and significantly better GPU utilization. 3. The performance validation of CoreWeave AI Object Storage, handling 40 GB/s writes and serving 110-120 GB/s reads for the 600 TB dataset. 4. Benchmarking results showing the efficiency gains of Ray Data over Ray Core for this specific workload.
  4. NVIDIA, CoreWeave, and AI Factory Operations at ScaleThis post details the operational aspects of running large-scale AI training and inference workloads, focusing on 'goodput' as the key economic driver. It elaborates on how CoreWeave's Mission Control and SUNK (Slurm on Kubernetes) software, combined with NVIDIA's hardware RAS features (Blackwell and Vera Rubin), contribute to achieving high goodput (up to 96%) and improved resiliency (10x MTBF improvement, 98% effective training time ratio). Specific technical contributions include: the measurement of Model FLOPs Utilization (MFU) and effective throughput, the role of hardware RAS engines in predictive maintenance and fault rerouting (e.g., NVLink routing in Vera Rubin), and the operational workflow of Mission Control processing over 200 million metrics samples per second for straggler detection and automated node draining. The post also highlights the industry-first bring-up and validation of NVIDIA Vera Rubin NVL72, emphasizing the continuous lifecycle of design, validation, and operation across hardware generations.
  5. CoreWeave SUNK adds new capabilities for modern AI research clusters | CoreWeave BlogThis post introduces new capabilities for CoreWeave SUNK, an industry-first unified training system for AI workloads. Key enhancements include SUNK self-service for faster cluster deployment and SUNK Anywhere to standardize training workflows across different infrastructure environments. The post also details improvements in observability with GPU straggler detection to identify performance bottlenecks in distributed training jobs. Automated User Provisioning (AUP) and SUNK User Provisioning (SUP) are introduced to streamline user and access management within SUNK clusters.
  7. GPU Stragglers and Training Failures ExplainedThis post details the failure modes of stragglers, synchronization stalls, and stalled GPUs in large-scale distributed AI training. It explains how these issues lead to inefficient compute utilization and proposes practical metrics like goodput, learning velocity, and MFU to quantify business impact. The post emphasizes that traditional infrastructure monitoring is insufficient and highlights the need for architectural control across placement, networking, storage, scheduling, and observability to prevent coordination failures. It positions CoreWeave Cloud as a platform designed to mitigate these issues by integrating these layers to reduce variance and improve training performance.
  8. How nTop Hit NASA’s 2030 CFD Goal Early on CoreWeave | CoreWeave BlogThis post details a joint proof of concept between nTop and CoreWeave where nTop Fluids, a GPU-native CFD solver, ran 10,000 Large-Eddy Simulations (LES) in 32 hours on CoreWeave's GPU cloud infrastructure. This achievement demonstrates the ability to run high-fidelity simulations at a massive scale, realizing NASA's CFD Vision 2030 goals four years early. The experiment utilized a Design of Experiments (DoE) methodology to explore 2,400 drone variants across five angles of attack. The success highlights the combination of scalable software and purpose-built infrastructure for high-volume, short-interval, high-fidelity simulations.
  9. Reference Architecture for AI Training | CoreWeave BlogThis post defines a reference architecture for distributed AI training, detailing four interdependent layers: topology-aware orchestration, checkpoint-optimized storage, high-performance interconnect, and integrated observability. It contrasts this purpose-built architecture with general-purpose infrastructure, highlighting how the former addresses failure modes like GPU stragglers, hangs, and slow recovery. The post emphasizes that resilience must be designed from the start, not added later, and that a purpose-built architecture leads to measurable improvements in utilization, stalled jobs, and recovery time, citing ~2x faster large-model training and improved sustained effective throughput.
  10. Where AI Training ROI Is Decided | CoreWeave BlogThis post discusses the critical bottlenecks in distributed AI training that impact ROI, specifically focusing on execution consistency across nodes, the difference between utilization and actual throughput (introducing Model FLOPs utilization - MFU), and networking/storage limitations. It argues that general-purpose clouds are not optimized for these distributed AI training challenges and highlights CoreWeave's purpose-built architecture, validated by SemiAnalysis ClusterMAX, MLPerf Training v5.0, and NVIDIA Exemplar Cloud status, as a solution for improving execution quality and translating capacity into measurable model progress.
  11. Why Distributed Training Fails at Scale | CoreWeave BlogThis post details the common failure modes in large-scale distributed training (compute failures, coordination failures, recovery failures) and the challenges in diagnosing them, particularly the prevalence of NCCL errors as surface symptoms. It highlights the significant impact of straggler nodes on job reliability and compute utilization, citing Meta FAIR research showing drastically reduced MTTF with increasing GPU cluster size and a ByteDance study indicating substantial GPU-hours wasted due to stragglers. The post emphasizes the need for infrastructure resiliency and deep observability to correlate telemetry with application behavior for accurate failure diagnosis.
  12. CoreWeave Powers GPT-NeoX-20B with EleutherAI and GooseAIThis post highlights the successful deployment and serving of EleutherAI's GPT-NeoX-20B model on CoreWeave Cloud. It emphasizes the accessibility and cost-effectiveness of the model for developers and researchers, and positions CoreWeave as a platform for managing such large-scale AI workloads, leveraging its GPU infrastructure and rapid auto-scaling capabilities for efficient serving.
  13. This post discusses MLOps innovations, specifically highlighting Kubeflow and MLflow as key tools for managing ML lifecycle management and training pipelines. It explains how these tools contribute to faster deployment times and increased efficiency in ML projects. The post also reiterates CoreWeave's infrastructure advantages for model training, including right-sizing workloads, bare-metal via Kubernetes, and its inference service. While it mentions MLOps and training pipelines, it does not introduce new technical details about CoreWeave's infrastructure for distributed training resiliency itself, but rather frames the benefits of CoreWeave's platform in the context of MLOps advancements.
  14. This post details the integration of Determined AI with CoreWeave's infrastructure to enable one-click deployment of large-scale training jobs. It highlights how CoreWeave's GPU resources, including A100s and A40s, combined with Determined AI's open-source ML platform, simplify the process of setting up, configuring, and managing AI clusters. The integration allows users to track and reproduce experiments, build more accurate models, and improve developer productivity, with a focus on cost-efficiency and speed compared to traditional cloud providers. The post also mentions the ability to run multi-GPU and multi-node distributed training out-of-the-box, supporting frameworks like GPT-Neo-X.
  15. 4 MLOps Best Practices for Efficiently Building AI Training Clusters | CoreWeaveThis post details four MLOps best practices for building efficient AI training clusters: 1. Regular node health checks to ensure optimal FLOP/s per dollar and identify/remediate unhealthy nodes, leveraging CoreWeave Mission Control for hardware verification, firmware updates, and management tests. 2. Continuous monitoring via model performance benchmarks for observability into node health, latency, and resource utilization, with CoreWeave Mission Control auto-remediating issues based on node conditions. 3. Fast and accurate data loading from disk to GPU using CoreWeave Tensorizer, which accelerates PyTorch tensor and model loading from HTTP/HTTPS and S3 endpoints via specialized serialization and zero-copy techniques. 4. Optimized orchestration using SUNK (Slurm on Kubernetes) for efficient resource utilization and streamlined deployment of Slurm jobs within Kubernetes clusters.
  16. 4 Ways to Strengthen AI Infra Resilience | CoreWeave BlogThis post details four practical strategies for enhancing AI infrastructure resilience: 1. Designing training code for failure tolerance and reproducibility (checkpointing, containerization, orchestration). 2. Implementing comprehensive instrumentation across hardware and software layers to differentiate infrastructure and application errors, correlating metrics like MFU with system-level data. 3. Automating alerts and recovery mechanisms, citing internal testing showing automated recovery is 3x faster than manual intervention and helps maintain high Effective Training Time Ratios (ETTR). 4. Stress-testing infrastructure with NCCL tests, burn-in/out tests, and controlled failure simulations to identify bottlenecks and validate recovery behavior before production workloads.
  17. A Slurm on Kubernetes Implementation for HPC and Large Scale AI | CoreWeaveThis post introduces SUNK (Slurm on Kubernetes), an implementation that integrates the Slurm workload manager with Kubernetes. It details how SUNK allows Slurm jobs to run within Kubernetes containers, enabling a unified platform for both HPC and large-scale AI workloads. The post explains the motivation for this integration (client efficiency and maximizing GPU utilization), the features of SUNK (containerized deployment via Helm, Kubernetes integration benefits, and state management), and the core components: Nodeset (a CRD for managing compute nodes), Syncer (for state synchronization between Slurm and Kubernetes), and a custom Kubernetes scheduler. It highlights the architectural diagram and the benefits of running Slurm components as containerized services within Kubernetes.
  18. CoreWeave, NVIDIA, and IBM Set MLPerf Record with Largest NVIDIA GB200 Blackwell Cluster, Achieving Over 2× Faster TrainingThis post details a new MLPerf Training v5.0 record set by CoreWeave, NVIDIA, and IBM using the largest-ever NVIDIA GB200 Blackwell cluster (2,496 GPUs). The submission achieved over 2x faster training performance than NVIDIA Hopper-based systems and demonstrated 91% scaling efficiency from 512 to 2,496 GPUs. The Carina cluster, running on CoreWeave's infrastructure, highlights the platform's ability to deliver and operate large-scale AI infrastructure at production level with high reliability. The results underscore the performance advantages of GB200 NVL72 for large-scale, multi-trillion parameter distributed training workloads and showcase CoreWeave's commitment to pushing the boundaries of AI infrastructure performance and scale.
  19. Decrease PyTorch Model Load Times with CoreWeave’s Tensorizer | CoreWeaveIntroduces CoreWeave Tensorizer, a tool for fast PyTorch module, model, and tensor serialization/deserialization. Tensorizer enables "zero-copy" model loading, significantly reducing model load times (under 5 seconds for models with billions of parameters) and memory usage by streaming tensors chunk-by-chunk directly to the GPU. This accelerates model instance spin-up times, reduces RAM requirements, and improves cost-efficiency for inference serving, especially during bursts in demand. It also offers S3/HTTP compatibility, sharding capabilities, and enhanced security by avoiding Python's pickle format.
  20. How To Get Better Workload Balancing | CoreWeaveThis post introduces SUNK (Slurm on Kubernetes) as a solution to unify training and inference workloads onto a single cluster, improving GPU infrastructure utilization and simplifying management. It highlights how SUNK enables flexible resource allocation between these traditionally separate workloads, leading to simplified scaling, faster time-to-market, and reduced operational overhead by eliminating the need for separate cluster environments. The post also emphasizes the ease of use and support provided for SUNK.
  21. MLPerf Results: A Record-Breaking, Cloud-Native AI Supercomputer | CoreWeaveThis post details CoreWeave's record-breaking MLPerf Training 3.0 submission, specifically the GPT-3 175B LLM benchmark, trained in under 11 minutes on over 3,500 NVIDIA H100 Tensor Core GPUs. It highlights the use of NVIDIA Quantum-2 InfiniBand networking and the HGX H100 cluster architecture, emphasizing the scale (over 3,500 GPUs, 4x larger than the next competitor) and speed (29x faster than the next best competitor). The post also mentions the cluster's use by Inflection AI for training their 'Pi' personal AI, showcasing the practical application of this infrastructure.
  22. NVIDIA H100 Benchmarks for Large-Scale Training | CoreWeaveThis post details a six-week benchmarking study on NVIDIA H100 GPUs at up to 1,024 scale, using a 30-billion parameter Llama 3-style architecture and 2 trillion tokens. It highlights infrastructure optimizations including bare-metal H100s, dual-fabric networking (InfiniBand for compute, DPU-offloaded Ethernet for storage), SUNK for topology-aware scheduling and automated re-queuing, and Tensorizer-based async checkpointing. The study achieved 51-52% MFU (vs. 35-45% typical), 3.66 days MTTF at 1,024 GPUs (10x improvement), and 97.5% ETTR. It also details custom tokenizer improvements and validates results against published configurations and NVIDIA DGX Cloud benchmarks.
  23. Unlocking cross-cloud AI | CoreWeave BlogThis post introduces SUNK Anywhere, extending CoreWeave's SUNK (Slurm on Kubernetes) system to support GPU environments across clouds or on-premises. This enables consistent cross-cloud AI training and inference by providing a unified control plane for scheduling, scaling, and running distributed workloads across CoreWeave, Google Cloud, AWS, or Azure. It addresses the complexity of traditional cross-cloud networking by offering a private, high-performance fiber-based network interconnect (CoreWeave Interconnect) directly connecting CoreWeave Cloud to Google Cloud, minimizing latency, ensuring reliability, and providing built-in security with MACsec encryption. The post also highlights the integration with Google Cloud's Partner Cross-Cloud Interconnect and the benefits of CoreWeave's global backbone network with no data transfer fees.
  24. What Is Node Lifecycle Management and Why Does It Matter for ML Training and Inference? | CoreWeave BlogThis post details CoreWeave's implementation of node lifecycle management for ML training and inference clusters. It introduces health checking mechanisms (passive and active) to monitor node performance and detect issues like GPU errors and temperature spikes. The post explains how these checks trigger automated actions, such as node reboots, draining, or RMAs, to maintain cluster health. It highlights the impact of node failures on training jobs, emphasizing the need for rapid detection and resolution due to the synchronous nature of these workloads. The post also touches upon the challenges of managing evolving hardware and the importance of observability in identifying root causes of failures.
  25. Why Your AI Cloud Training Strategy Is FailingThis post details a large-scale pretraining benchmark for a 30-billion-parameter LLM across 1,024 GPUs, measuring infrastructure performance under extreme load. Key results include achieving 51-52% Model FLOPs Utilization (MFU), 97-98% Effective Training Time Ratio (ETTR), and a 10x improvement in Mean Time to Failure (MTTF) to 3.66 days compared to the industry benchmark of 0.33 days. These metrics demonstrate that CoreWeave's purpose-built AI cloud infrastructure, with its vertically integrated stack, bare-metal access, dual network fabrics, topology-aware orchestration, high-speed data pipelines, and deep observability, enables significantly faster time-to-market (7-15 days faster for a 30-day training cycle) and improved reliability for large-scale AI training workloads.
  26. SUNK: Production-Grade AI Training at ScaleThis post introduces SUNK (Slurm on Kubernetes), a new unified training system that integrates Slurm with Kubernetes to provide production-grade AI training at scale. Key contributions include topology-aware scheduling, automated health management and failure mitigation, deep operational visibility, and unified lifecycle management. It highlights improvements in goodput (up to 96%), Mean Time to Failure (10x longer for thousand-GPU clusters), and job restart times (~90 seconds). The post also mentions SUNK User Provisioning (SUP) for automated user onboarding and permissions, and integration with CoreWeave Mission Control for continuous health management and observability.