ML Inference Benchmarking

7/30/2026

This post details the selection criteria and mapping of various NVIDIA GPU platforms (GB300 NVL72, GB200 NVL72, HGX B300, HGX B200, HGX H200, HGX H100, RTX PRO 6000 Blackwell Server Edition) to specific AI inference workload patterns. It provides technical justifications for choosing certain GPUs based on factors like model size, context window, concurrency, batching, latency, and deployment topology, referencing performance data and architectural features (e.g., NVLink bandwidth, Transformer Engine, KV cache bottlenecks). It also outlines CoreWeave's integrated inference solutions.

7/30/2026

This post details CoreWeave's achievement of NVIDIA Exemplar Cloud validation for inference on the NVIDIA GB200 NVL72 platform. It highlights the successful execution of NVIDIA's inference benchmarks across Reasoning, Chat, Summarization, Generation, and Disaggregation phases for DeepSeek-R1, Llama 3.3, and GPT-OSS models. The post emphasizes the role of CoreWeave Mission Control and its GPU Straggler Detection feature in monitoring and optimizing collective metrics, ensuring high throughput and low Time-to-First-Token (TTFT) latency, meeting or exceeding NVIDIA's performance standards.

7/30/2026

This post introduces CoreWeave's leading results in the MLPerf 0.7 Endpoints benchmark, showcasing their ability to achieve high throughput and efficiency with large language models like DeepSeek-R1 on NVIDIA GB200 NVL72 systems. It quantifies performance metrics such as sustained output tokens per second and tokens per second per GPU, and explains the significance of these metrics for inference services in terms of cost and user experience. The post also details the specific infrastructure components (bare metal, networking, CKS, SUNK, DFS) that contribute to these benchmark achievements, emphasizing that testing was conducted on production-ready infrastructure.

7/30/2026

This post details the engineering work involved in optimizing the Kimi K2.7 Code model for inference on CoreWeave's Blackwell GPU infrastructure. Key contributions include: implementing NVFP4 quantization by converting INT4 weights using NVIDIA Model-Optimizer with a custom 3-pass calibration (short context, long context math, long context coding), training a DFlash speculative decoding model using a generated dataset and the D-PACE loss function for improved acceptance length, and contributing support for D-PACE to the TorchSpec project and for AL evals to AIPerf. The entire process was managed on CoreWeave Kubernetes Service with RoCE fabric for fast training, and the optimized model is deployed on CoreWeave Inference via vLMM.

7/30/2026

This post details CoreWeave's MLPerf® Training v6.0 records, specifically achieving industry-leading time-to-train for DeepSeek-V3 671B in 2.02 minutes using an 8,192-GPU NVIDIA GB300 NVL72 cluster with NVIDIA Spectrum-X Ethernet networking. It highlights efficient performance and scale across various cluster sizes (64 to 8,192 GPUs) and models (DeepSeek-V3 671B, Llama-3.1-405B, GPT-OSS-20B, Llama 3.1 8B). The post emphasizes the full-stack engineering approach, including NVLink-domain-aware scheduling in CoreWeave Kubernetes Service (CKS) and topology-aware workload placement in SUNK, deep networking optimizations, and fleet-wide performance consistency managed by CoreWeave Mission Control. It also details the use of NVIDIA NeMo Framework Release 26.04, full CUDA Graphs, and tuned parallelism strategies.

7/30/2026

This post details CoreWeave's MLPerf 6.0 leadership, showcasing doubled server mode throughput for DeepSeek R1 with NVIDIA GB300 NVL72 compared to MLPerf 5.1. It highlights leading offline mode throughput for DeepSeek R1 on GB300 NVL72 and improved throughput for GPT-OSS-120B on GB300 NVL72 compared to GB200 NVL72. The post also introduces new inference product offerings: Serverless Inference (via W&B), Dedicated Inference (preview), and Inference on CKS, emphasizing a shared architectural base for consistent performance and cost visibility across different operational needs.

7/30/2026

This post details the causes and measurement strategies for inference latency and availability drift in production. It identifies three primary sources of latency drift: infrastructure-layer variability, model-serving configuration drift (batching, KV cache pressure), and traffic pattern mismatch. It also explains how availability degrades through autoscaling lag, request shape mismatches, and stale cache behavior. The post emphasizes the need to measure metrics like p99 latency, TTFT, goodput, error budget consumption, autoscaling lag, and timeout rates under load to diagnose and prevent these issues, which often go unnoticed by standard monitoring.

7/30/2026

This post details the strategic importance and technical implications of the llm-d project moving into the CNCF Sandbox. It explains how llm-d addresses the unique challenges of production inference at scale, such as statefulness, hardware sensitivity, and cost-efficiency, by introducing a purpose-built orchestration layer. The post highlights llm-d's integration with Kubernetes-native components like KServe and Gateway API, and its role in transforming distributed inference into a manageable, observable cloud-native workload. CoreWeave's contribution is framed as providing real-world operational insights to the open-source project.

5/20/2026

This post details CoreWeave's strategies for achieving up to 96% AI infrastructure goodput. It emphasizes the importance of GPU reliability and the challenges in training large models due to scaling laws and accelerated model launch times. The post outlines three key strategies: 1. Utilizing high-performance hardware optimized for AI workloads, including NVIDIA GB200 NVL72 systems, NVIDIA Quantum-2 InfiniBand, NVIDIA BlueField-3 DPUs, CoreWeave Kubernetes Service (CKS), and Slurm on Kubernetes (SUNK) for topology-aware scheduling. 2. Proactively identifying and remediating interruptions through CoreWeave Mission Control, which provides advanced cluster validation, health monitoring, proactive node replacement, and deep observability. This includes the Fleet Lifecycle Controller for rigorous AI infrastructure validation and the Node Lifecycle Controller for continuous monitoring and proactive health checks to detect and replace unhealthy nodes. 3. Providing customers with full transparency into cluster health and performance, enabling measurement of detailed hardware metrics (GPU health, fan speed, temperature) and job-level metrics for faster diagnosis and resolution of interruption issues. The post also highlights the 24/7 monitoring by FleetOps and CloudOps teams and close collaboration with customers.

5/20/2026

This post details CoreWeave's achievement of new industry benchmarks for AI training workloads on NVIDIA GB200 NVL72 systems, becoming the first NVIDIA Exemplar Cloud for this hardware. It quantifies performance improvements using Model Flops Utilization (MFU) across several models (DeepSeek v3, Grok-1 314B, Llama 3.1 405B, Llama 3 70B), consistently exceeding NVIDIA's reference targets. The post highlights the role of CoreWeave Mission Control in operating the cluster with efficiency and reliability, including predictive failure detection and proactive component replacement. It also details the integrated software stack, including CoreWeave Bare Metal, CKS, and SUNK, and the use of observability tools like Cabinet Visualizer and Weights & Biases for monitoring and demonstrating straggler detection.

5/20/2026

This post introduces CoreWeave's MLPerf Inference v5.0 results, showcasing performance with NVIDIA GB200 Grace Blackwell Superchips (800 TPS on Llama 3.1 405B, 2.86X per-chip improvement over H200) and H200 GPUs (33,000 TPS on Llama 2 70B, 40% improvement over H100). It details the infrastructure optimizations enabling these results, including bare-metal Kubernetes, topology-aware scheduling with SUNK, and model loading acceleration with Tensorizer.

5/20/2026

This post announces the general availability of NVIDIA HGX B200 instances on CoreWeave, expanding the NVIDIA Blackwell fleet. It details the technical specifications of these instances, including 8 NVIDIA B200 Blackwell GPUs, NVIDIA NVLink and NVLink Switch, Intel Emerald Rapids CPUs, NVMe local storage, NVIDIA BlueField-3 DPU, and NVIDIA ConnectX-7 InfiniBand HCAs for a 400G NDR fabric. It highlights the integration with CoreWeave SUNK for unified training and serving, and CKS for bare-metal performance. It also mentions CoreWeave Observe™ for per-GPU monitoring and Mission Control for reliability.

5/20/2026

Introduces the general availability of NVIDIA RTX PRO 6000 Blackwell Server Edition instances, the first cloud provider to offer this GPU architecture. Highlights performance gains for LLM inference and text-to-video generation, detailing the hardware specifications (GDDR7 memory, Tensor Cores, RT Cores) and system configurations (up to 8 GPUs, Intel Emerald Rapids CPUs, NVIDIA BlueField-3 DPUs). Mentions integration with CoreWeave's Observability services, CAIOS, and LOTA, and availability via CKS and SUNK.

5/20/2026

This post details CoreWeave's achievement of leading inference performance for the Kimi K2.6 model, as benchmarked by Artificial Analysis. It highlights the technical optimizations employed, including training a custom NVFP4 quantized model and implementing EAGLE3 speculative decoding on NVIDIA GB300 and GB200 NVL72 clusters. The post also mentions the validation of these optimizations across various benchmarks and the integration of these performance enhancements throughout the CoreWeave Inference stack, from hardware access to custom tuning.

5/20/2026

This post details CoreWeave's achievement of the highest Platinum rating in SemiAnalysis's ClusterMAX™ Rating System, highlighting their leadership in operating large-scale 10k+ H100 clusters reliably. It elaborates on the purpose-built nature of the CoreWeave Cloud Platform for AI workloads, including infrastructure, managed software, and application services, augmented by Mission Control and Observability software. The post emphasizes proprietary software for provisioning, orchestration, and monitoring to ensure high availability and minimize downtime. It also details CoreWeave's commitment to being first to deploy the latest GPU technologies at scale and their continuous improvement loop based on internal data and learnings. Specific differentiators discussed include security, reliability, and SLAs, with mention of industry best practices and standards. Performance optimizations across infrastructure, managed Kubernetes, and application services are highlighted, leading to higher GPU cluster performance and goodput. The post also mentions proactive health checks and automated node lifecycle management for reliability.

5/20/2026

This post details CoreWeave's achievement of new industry records in MLPerf Training v5.0 benchmarks, showcasing the performance of their largest-ever NVIDIA Blackwell GPU cluster. It highlights specific performance gains, such as training Llama 3.1 405B in 27.3 minutes, more than twice as fast as comparable Hopper GPU clusters. The post elaborates on the underlying infrastructure components that contribute to this performance, including AI-optimized data centers, performance-optimized GPU instances, bare metal access, high-performance storage (up to 2GB/s/GPU), NVIDIA Quantum InfiniBand and BlueField DPUs, SUNK (Slurm on Kubernetes), CoreWeave Kubernetes Service (CKS), and Mission Control for observability and uptime. It quantifies customer benefits like up to twice as fast training speeds, 20% higher performance on like-for-like clusters, and a 14% improvement in price-adjusted performance.

5/20/2026

This post details CoreWeave's record-breaking MLPerf Training v5.0 submission using the largest NVIDIA GB200 Blackwell cluster (2,496 GPUs), achieving over 2x faster training performance than comparable H100 systems. It highlights the infrastructure density benefits of GB200 NVL72, reducing physical space requirements by 4x. The post also emphasizes the reliability and scaling efficiency of the Carina cluster, demonstrating CoreWeave's capability in delivering and operating large-scale AI infrastructure at production level.

5/20/2026

This post introduces and compares three primary approaches to building AI models: pretraining, fine-tuning, and Retrieval-Augmented Generation (RAG). It details the pros, cons, costs, and time investments associated with each approach, offering guidance on selecting the most suitable method for different project needs. The post also provides real-world examples and discusses hybrid strategies, including the use of techniques like LoRA, QLoRA, multi-hop retrieval, and query decomposition.

5/20/2026

This post announces an expansion of CoreWeave's collaboration with NVIDIA, focusing on building AI factories and integrating NVIDIA's latest hardware (GPUs, Vera CPUs, BlueField storage) into their infrastructure. It highlights the company's commitment to scaling AI compute and making AI usable and productive across industries, emphasizing the shift from possibility to production. The post also mentions CoreWeave's software offerings like Mission Control and SUNK as defining standards for AI cloud solutions.

5/12/2026

Introduces a deployment blueprint for running Red Hat AI Inference on CoreWeave Kubernetes Service (CKS) to enable hybrid inference across on-premises and cloud environments. This reference architecture allows enterprises to run the same open-source inference stack on-premises and on CoreWeave, leveraging Kubernetes-native control and open runtimes. It complements CoreWeave's existing inference portfolio by offering a supported path for self-managed inference on CKS using Red Hat's stack, building on the collaboration around the llm-d project.

4/15/2025

This post announces the launch and scaling of NVIDIA GB200 NVL72 systems, highlighting CoreWeave's rapid operationalization of advanced AI infrastructure. It references previous achievements and benchmark results (MLPerf v5.0) to underscore performance and efficiency, and details the integration with CoreWeave's existing cloud services and networking capabilities.

4/2/2025

This post announces CoreWeave's record-breaking AI inferencing benchmark results using NVIDIA GB200 Grace Blackwell Superchips, achieving 800 TPS on Llama 3.1 405B. It also reports a 40% throughput increase for Llama 2 70B on H200 GPUs compared to H100. The post emphasizes CoreWeave's role as the first cloud provider to offer GB200 NVL72 instances and highlights their prior achievements in deploying H100 and H200 GPUs, and demoing GB200 NVL72 systems.