TechPrufer
Get started
Coreweave
Coreweave logoCoreweave · Feature trail

ML Inference Benchmarking

47 posts · Apr 2025 – Sep 2026

What this trail covers

CoreWeave enhances its AI infrastructure for scaling complex workloads, focusing on production inference and training performance. This post details the selection of NVIDIA GPUs for inference workloads, mapping specific GPU platforms (GB300 NVL72, GB200 NVL72, HGX B300, HGX B200, HGX H200, HGX H100, RTX PRO 6000 Blackwell Server Edition) to common inference patterns based on model size, context window, concurrency, batching behavior, latency SLOs, and deployment shape. It highlights the comparison of MLPerf® Inference v6.1 results across various models and NVIDIA platforms, demonstrating leading performance in throughput and efficiency for multimodal, MoE, and large language models on production clusters.

All Coreweave trails
  1. MLPerf® Inference v6.1 Results: CoreWeave Leads ProvidersThis post details CoreWeave's MLPerf® Inference v6.1 results, showcasing leading performance on NVIDIA Blackwell and Blackwell Ultra platforms across multimodal (Qwen3-VL-235B-A22B), MoE (DeepSeek-R1-671B), and large language models (GPT-OSS-120B, Llama 2 70B). It highlights specific throughput gains, such as a nearly 20% increase in per-GPU throughput for DeepSeek-R1 over v6.0, and the highest per-GPU throughput for GPT-OSS-120B. The results were achieved on production clusters utilizing CoreWeave Mission Control for monitoring and SUNK for topology-aware scheduling within NVLink domains.
  2. Open Models Can Win on Cost and Control | CoreWeave BlogThis post contributes to the ML Inference Benchmarking thread by shifting the focus from pure performance benchmarks to a cost and control analysis of open-weight models versus closed-source models. It introduces the concept that for many production workloads, 'good enough' performance from open models is sufficient, and the primary differentiator becomes cost and control. It details the cost savings achievable with open models (e.g., $1/million tokens vs. $50/million for closed models) and the advantages of self-hosting and fine-tuning for control and customization. The post also highlights the critical role of robust infrastructure in realizing these cost savings, implicitly reinforcing the need for CoreWeave's platform.
  3. Deploy Kimi K3 Dedicated Inference on CoreWeaveThis post details the deployment of Kimi K3, a 2.8 trillion-parameter open-source model, on CoreWeave's Dedicated Inference platform using NVIDIA GB300 NVL72 systems. It provides performance benchmarks comparing GB300 and GB200 NVL72 for Kimi K3, focusing on output token throughput, mean request latency, and p50 decode throughput per user under agentic workloads. The post also describes the underlying stack, including NVIDIA Dynamo and vLLM with specific optimizations like KV-cache aware routing and speculative decoding, to achieve production-scale inference for large models.
  4. Choosing the Right NVIDIA GPU for Running Inference | CoreWeave BlogThis post provides a detailed mapping of various NVIDIA GPU platforms (GB300 NVL72, GB200 NVL72, HGX B300, HGX B200, HGX H200, HGX H100, RTX PRO 6000 Blackwell Server Edition) to specific inference workload profiles. It outlines how factors like model size, context window, concurrency, batching behavior, latency SLOs, and deployment shape influence the choice of GPU. The post highlights the performance benefits and cost efficiencies of different platforms for use cases ranging from frontier-scale MoE models and high-concurrency chat endpoints to agentic AI inference and multimodal serving. It also introduces CoreWeave's unified inference offerings (CoreWeave Inference, Inference on CKS, Dedicated Inference, Serverless Inference) as solutions for managing these diverse inference needs.
  5. CoreWeave Earns NVIDIA Exemplar Validation for GB200CoreWeave achieves NVIDIA Exemplar Cloud validation for inference on the NVIDIA GB200 NVL72 platform, demonstrating high inference throughput and latency results that meet NVIDIA's reference architecture standards. The post details the specific inference benchmark tests conducted (Reasoning, Chat, Summarization, Generation, Disaggregation) across DeepSeek-R1, Llama 3.3, and GPT-OSS models, measuring TPS/GPU and Time-to-First-Token (TTFT) latency. It highlights the use of NVIDIA Dynamo for multi-node inference and TRT-LLM as the backend. The post also emphasizes the role of CoreWeave Mission Control and its GPU Straggler Detection feature in monitoring and ensuring performance and reliability during these benchmarks.
  7. CoreWeave Leads MLPerf 0.7 Endpoints | CoreWeave BlogCoreWeave achieved leading results in the MLPerf 0.7 Endpoints benchmark by running the DeepSeek-R1 model on 68 NVIDIA Blackwell GPUs in a GB200 NVL72 system. The post details sustained system throughput of 441,740 output tokens per second at 16,384 concurrent requests and up to 6,496 tokens per second per GPU. It emphasizes that per-GPU throughput and time per output token (TPOT) are critical metrics for inference services, influencing both user experience and cost. The benchmark was run on standard production infrastructure, including bare-metal servers, NVIDIA Quantum-2 InfiniBand networking, CKS, SUNK for topology-aware scheduling, and DFS for high-throughput storage, demonstrating the platform's capability for real-world inference workloads.
  8. CoreWeave Leads on Artificial Analysis for Kimi K2.7 Code Inference | CoreWeave BlogThis post details the engineering work involved in optimizing Kimi K2.7 Code for production inference on CoreWeave's infrastructure. Key contributions include: - Metal-to-model optimization: Applying NVFP4 quantization to leverage Blackwell's FP4 execution and integrating a DFlash speculative decoder. - Quantization process: Using NVIDIA's Model-Optimizer with a custom 3-pass calibration on general conversation, math, and coding datasets. - Speculative decoding training: Generating a new dataset using the NVFP4 checkpoint and training a DFlash speculative decoding model with the D-PACE loss function. - Benchmarking: Utilizing AIPerf from the NVIDIA Dynamo team and contributing support for common AL evals to measure acceptance length and compare results. - Infrastructure integration: Training occurred on GB300 NVL72 racks managed by CoreWeave Kubernetes Service over a RoCE fabric. Production endpoint uses NVFP4 + DFlash on vLLM by default. - Performance validation: Achieving leading price-performance as validated by Artificial Analysis.
  9. CoreWeave Sets MLPerf® Training v6.0 Records | CoreWeave BlogCoreWeave sets new MLPerf® Training v6.0 records for DeepSeek-V3 671B (2.02 minutes with 8,192 NVIDIA GB300 NVL72 GPUs) and Llama-3.1-405B (9.77 minutes with 4,096 Blackwell Ultra GPUs). The post details the infrastructure optimizations that enable efficient scaling across thousands of GPUs, including NVLink-domain-aware scheduling in CKS and SUNK, topology-aware workload placement, and deep networking optimizations using NVIDIA Spectrum-X Ethernet with RoCE. It highlights the importance of system-wide optimization beyond raw compute capacity for real-world performance and economics, and demonstrates consistent scaling efficiency on smaller 64-GPU clusters using NVIDIA HGX B200 with InfiniBand.
  10. CoreWeave's Innovation Velocity Drives in MLPerf 6.0 LeadershipThis post details CoreWeave's performance leadership in MLPerf 6.0, specifically highlighting throughput gains on NVIDIA GB300 NVL72 and GB200 NVL72 systems for models like DeepSeek R1 and GPT-OSS-120B. It quantifies performance improvements, such as doubling throughput in server mode for DeepSeek R1 on GB300 NVL72 compared to MLPerf 5.1, and a 1.6x improvement in offline mode. It also shows a 14% improvement in server mode and 34% in offline mode for DeepSeek R1 when comparing GB300 NVL72 to GB200 NVL72, and up to 28% higher throughput for GPT-OSS-120B in server mode when comparing GB300 NVL72 to GB200 NVL72. The post also elaborates on CoreWeave's inference product offerings (Serverless, Dedicated, and CKS) and their shared architectural foundation.
  11. GLM 5.2 on CoreWeave Inference | CoreWeave BlogThis post announces the availability of GLM 5.2 on CoreWeave Inference, highlighting its strong performance and price-performance benchmarks compared to other providers. It details CoreWeave's engineering efforts in tuning GLM 5.2 for their hardware, optimizing the full inference stack (GPUs, networking, model/container loading, KV cache, serving runtime) to achieve fast inference speeds and cost-efficiency. The post emphasizes the benefits of using managed endpoints for open-weight models, allowing users to focus on application development rather than infrastructure management.
  12. Introducing NVIDIA HGX B300 on the Essential Cloud for AIThis post introduces the availability of NVIDIA HGX B300 on CoreWeave Cloud, highlighting its performance improvements over the HGX B200, particularly for agentic AI workloads. Key technical details include: doubled interconnect speed with NVIDIA Quantum-X800 InfiniBand, BlueField-3 DPUs, and 800 Gbps NVIDIA ConnectX-8 SuperNICs; 50% increase in GPU memory capacity over B200; benchmarks showing 3.42x faster token generation on Kimi K2.5 and 4.93x faster end-to-end request latency on DeepSeek-R1 (vs. NVIDIA HGX H200). It details the HGX B300's NVFP4 performance increase from 9 PFLOPs to 14 PFLOPs, and system-level specifications like 1.8 TB/s per-GPU interconnect bandwidth and 7.7 TB/s memory bandwidth. The post also touches on the integration with CoreWeave's infrastructure for provisioning, orchestration, observability, and data access, emphasizing the importance of low-latency, high-bandwidth interconnects for agent coordination and sustained GPU utilization.
  13. Production AI Runs on Inference | CoreWeave BlogThis post emphasizes the critical role of inference in production AI, moving beyond proof-of-concept to real-world requirements of latency, reliability, and cost. It highlights the evolution of inference demands driven by agentic workflows and interactive applications. The post details the need for production-grade inference to provide reliability through infrastructure and visibility, cost predictability through appropriate economic models (token-based vs. dedicated capacity), and control that evolves with the workload. It introduces CoreWeave Inference with three deployment paths: Serverless Inference (API-based, auto-scaling, token pricing), Dedicated Inference (predictability, customization, BYOM), and Managed Inference (fully managed, optimized for specific hardware). It references hardware advancements like NVIDIA Vera Rubin NVL72 for improved inference per watt and lower token cost, and discusses the importance of matching infrastructure economics to workload behavior.
  14. The Token Pricing Illusion: AI Inference Costs | CoreWeave BlogThis post introduces a diagnostic metric for 'cost per useful token' to address the limitations of token pricing for AI inference. It defines a 'useful token' based on latency SLOs, context relevance, and single billing. The post analyzes three ways sticker price abstracts reality: SLO misses, idle capacity, and autoscaling overhead. It then categorizes four production inference workload patterns (Exploration & Iteration, Variable Production, Steady-State Production, Agentic at Scale) and recommends either token pricing or GPU-billed pricing based on utilization and performance requirements. This provides a framework for customers to evaluate their inference economics beyond simple per-token costs.
  15. Why Inference Is the Defining Layer of AI | CoreWeaveThis post shifts the focus from training to inference as the defining layer of AI, highlighting the projected market growth and the increasing complexity introduced by agentic AI. It details the operational challenges of production inference, including GPU reservation, unpredictable I/O, high token generation, downstream stalling, and security concerns. The post contrasts benchmark performance with real-world production behavior, attributing discrepancies to system-level issues like cold starts, traffic spikes, observability gaps, and cost drift. It outlines CoreWeave's full-stack infrastructure approach, emphasizing bare-metal GPUs, high-speed networking, optimized object storage, and managed deployment options to address these challenges. The post also introduces the concept of 'Burst Compute' for on-demand GPU scaling and discusses the importance of accessing new GPU generations first for economic advantage.
  16. Why Inference Latency, Availability Drift | CoreWeave BlogThis post details the challenges of inference latency and availability drift in production, focusing on the gradual degradation that often goes unnoticed by standard monitoring. It identifies three primary causes for latency drift: infrastructure-layer variability (GPU contention, scheduling overhead), model-serving configuration drift (batching, KV cache pressure), and traffic pattern mismatch (autoscaling lag, request shape). The post introduces key metrics for diagnosing latency drift, including p99 latency, time-to-first-token (TTFT), and goodput vs. throughput. It further explains how availability degrades without outright failure through elevated error rates, autoscaling lag, and stale cache behavior, proposing metrics like error budget consumption rate, autoscaling lag vs. burst profile, and timeout rate under load to monitor availability degradation. The core contribution is a framework for understanding and measuring the subtle, drift-based failures common in large-scale inference systems.
  17. Why llm-d in CNCF Matters for Production Inference | CoreWeave BlogThis post details the strategic importance and technical implications of the llm-d project moving into the CNCF Sandbox. It explains how llm-d addresses the unique challenges of production inference at scale, such as statefulness, hardware sensitivity, and cost-efficiency, by introducing a purpose-built orchestration layer. The post highlights llm-d's integration with Kubernetes-native components like KServe and Gateway API, and its role in transforming distributed inference into a manageable, observable cloud-native workload. CoreWeave's contribution is framed as providing real-world operational insights to the open-source project.
  18. First-Ever Measured Vera Rubin NVL72 Silicon Performance Stats | CoreWeave BlogThis post provides the first measured silicon performance statistics for the NVIDIA Vera Rubin NVL72 on CoreWeave, specifically comparing its energy efficiency (tokens per megawatt) against NVIDIA Blackwell NVL72 using the DeepSeek R1 model. It details the inference optimizations used (expert parallelism, NVFP4 precision, multi-token prediction, disaggregated prefill/decode via TensorRT-LLM and Dynamo) and quantifies the performance gains in terms of tokens per second per megawatt at matched interactivity targets. The post also discusses the operational layer supporting Vera Rubin NVL72, including liquid cooling and rack control, and anticipates further performance improvements through ongoing software stack optimization.
  19. This post details AI Dungeon's migration to CoreWeave's Inference Service, highlighting a reduction in compute instance spin-up times to 45 seconds, minimization of idle compute, and a 76% reduction in computing costs. It also notes a 50% drop in inference latencies using CoreWeave's Inference Optimized V100 instances, leading to a better in-game experience for users and reduced resource planning for AI Dungeon.
  20. CoreWeave announces the deployment of the largest fleet of NVIDIA A40 GPUs in North America. The A40 GPU offers 2x rendering performance and 3x AI training performance over the previous generation, with 48 GB of graphics memory, accelerating photorealistic rendering, professional graphics applications, large-scale display experiences, virtual sets, VR, broadcast-grade streaming, and AI/data science model training.
  21. CoreWeave Powers GPT-NeoX-20B with EleutherAI and GooseAIThis post details the performance of EleutherAI's GPT-NeoX-20B model, highlighting its accessibility and cost-effectiveness compared to other models. It provides a performance comparison table for factual question answering and language modeling tasks, demonstrating GPT-NeoX-20B's superiority. The post also outlines two ways to get started with the model: via a managed API on GooseAI or by self-managing infrastructure on CoreWeave Cloud, emphasizing CoreWeave's GPU offerings, spin-up times, and auto-scaling capabilities.
  22. This post details the advantages of using GPT-J, an 6B parameter open-source NLP model, on CoreWeave's infrastructure. It highlights CoreWeave's fast spin-up times, responsive autoscaling (including scale-to-zero), and competitive pricing for serving GPT-J. Customer examples from AI Dungeon and NovelAI demonstrate significant cost savings (75%) and latency reductions (50% and 3x faster serving respectively). The post also mentions CoreWeave's collaboration with EleutherAI on a DGX A100 training cluster and the availability of CoreWeave engineers for DevOps optimization.
  23. This post details the deployment of the BLOOM-176B LLM as an InferenceService on CoreWeave. It highlights the complexity of serving large LLMs and positions CoreWeave's infrastructure as a solution for reducing this complexity. The post also mentions CoreWeave's A100 distributed training clusters, NVIDIA Quantum InfiniBand networking, and KNative Serving for auto-scaling, emphasizing performance and cost-efficiency.
  24. This post details CoreWeave's infrastructure for training, fine-tuning, and serving Large Language Models (LLMs). It highlights the use of bare-metal hardware, NVIDIA Quantum InfiniBand networking with NVSWITCH and NVIDIA SHARP for distributed training, and NVLINK for fine-tuning. For serving, it mentions KNative Serving on Kubernetes for auto-scaling and fast spin-up times, and provides a comparison table of various LLMs (e.g., BLOOM 176B, GPT-J 6B) based on parameters, context window, and use cases. It also emphasizes cost-effectiveness and performance-adjusted costs.
  25. This post details the performance gains achieved by CoreWeave's new NVIDIA GB300 NVL72 instances, featuring NVIDIA Blackwell Ultra GPUs, when running the DeepSeek R1 reasoning model. It quantifies a 6.5x performance improvement on inference compared to a 16-GPU NVIDIA H100 system, attributing this to the GB300's superior memory and interconnect bandwidth, which allows for 4-way Tensor Parallelism (TP4) versus TP16 on H100, significantly reducing communication overhead. The post also elaborates on the GB300 NVL72 architecture, including its 37TB memory, 130TB/s NVLink bandwidth, and integration with NVIDIA Quantum-X800 InfiniBand. CoreWeave's platform-specific optimizations, such as the Rack LifeCycle Controller, topology-aware scheduler leveraging NVLink labels, and observability through Grafana dashboards, are presented as key enablers of this record performance.
  26. Achieve AI Infrastructure Goodput of up to 96% with 3 Key StrategiesThis post details three strategies CoreWeave employs to achieve high AI infrastructure goodput (up to 96%): 1. Utilizing high-performance hardware optimized for AI workloads, including the latest NVIDIA GPUs (GB200 NVL72), NVIDIA Quantum-2 InfiniBand, and NVIDIA BlueField-3 DPUs, along with a purpose-built software stack like CoreWeave Kubernetes Service (CKS) and Slurm on Kubernetes (SUNK) for topology-aware scheduling. 2. Proactively identifying and remediating interruptions through rigorous health checking and monitoring via CoreWeave Mission Control, Fleet Lifecycle Controller, and Node Lifecycle Controller, which detect subtle issues like silent data corruption and proactively swap unhealthy nodes. 3. Optimizing resource utilization through efficient scheduling and monitoring, providing customers with transparency into hardware metrics and detailed job-level metrics. The post emphasizes the importance of minimizing interruptions and enabling quick recovery for faster time-to-market and cost-effective model training and deployment.
  27. Best GPUs for Rendering | CoreWeaveThis post provides guidance on selecting the best NVIDIA RTX professional GPUs for rendering workloads on the Conductor platform. It details considerations for GPU type, number of GPUs, and price-performance tuning, and lists available GPU SKUs (Quadro RTX 4000, Quadro RTX 5000, RTX A4000, RTX A5000, RTX A6000).
  28. Coming Soon: NVIDIA RTX PRO 6000 Blackwell Server Edition on CoreWeave | CoreWeaveThis post announces the upcoming availability of NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs on CoreWeave. It highlights the performance improvements over the L40S, including 5x AI performance at FP4 precision and 2x rendering performance. Specific metrics are provided for LLM inference (up to 5.3x faster) and text-to-image generation (up to 3.5x faster). The post also details the integration of these GPUs with Intel Emerald Rapids CPUs and NVIDIA BlueField DPUs for enhanced security and networking, and mentions support for CoreWeave Kubernetes Service (CKS), Slurm on Kubernetes (SUNK), Observability Services, AI Object Storage (CAIOS), and Local Object Transport Accelerator (LOTA).
  29. CoreWeave Becomes First NVIDIA GB200 Exemplar CloudCoreWeave details its achievement as the first NVIDIA GB200 Exemplar Cloud, demonstrating performance exceeding NVIDIA's training targets on GB200 NVL72 systems. The post quantifies this by reporting higher Model Flops Utilization (MFU) compared to reference targets for DeepSeek v3 (1.9% greater), Grok-1 314B (4.7% greater), Llama 3.1 405B (3.8% greater), and Llama 3 70B (2.4% greater) on 512 GPUs. It highlights the integration of CoreWeave Mission Control for predictive failure detection, bare-metal servers for low-latency access, CKS for runtime environment, and SUNK for topology-aware scheduling. Observability through CoreWeave Cabinet Visualizer and Weights & Biases is also emphasized for monitoring and detecting underperforming components.
  30. CoreWeave Delivers Breakthrough AI Performance with NVIDIA GB200 and H200 GPUs in MLPerf Inference v5.0This post details CoreWeave's MLPerf Inference v5.0 submission results, highlighting breakthrough performance with NVIDIA GB200 Grace Blackwell Superchips and H200 GPUs. It quantifies performance gains: 800 TPS on Llama 3.1 405B with GB200 (2.86X per-chip speedup over H200) and 33,000 TPS on Llama 2 70B with H200 (40% improvement over H100). The post explains how CoreWeave Cloud Platform optimizations, including bare-metal infrastructure, advanced networking, Mission Control for resource management, CKS for inference engines, SUNK for efficient scheduling, and Tensorizer for model loading, contribute to these results. It also provides specific benchmark numbers and comparisons against previous generations and competitors.
  31. CoreWeave Drives 20% Higher GPU Cluster Performance | CoreWeaveThis post quantifies CoreWeave's AI infrastructure efficiency by demonstrating up to 20% higher Model FLOPS Utilization (MFU) compared to public benchmarks for Llama 3.1 training runs on 128 NVIDIA H100 GPUs. It shows MFU of 51.9% (a 28% increase over Aleph Alpha's A100 results) and 49.2% (an 18% increase over MosaicML's H100 results). The post attributes these gains to optimizations across the stack, including bare-metal Kubernetes, high-performance networking (InfiniBand and Spectrum switches), optimized asynchronous checkpointing (Tensorizer), and high-speed object storage (2 GB/s per GPU). It also mentions parity with NVIDIA's DGX Cloud Benchmarking Recipes.
  32. CoreWeave Expands its NVIDIA Blackwell Fleet with Generally Available NVIDIA HGX B200 InstancesThis post announces the general availability of NVIDIA HGX B200 instances on CoreWeave, expanding the Blackwell fleet. It details the hardware specifications of these instances, including 8 NVIDIA B200 Blackwell GPUs, 2 Intel Emerald Rapids CPUs, 61.44TB NVMe local storage, and NVIDIA Quantum-2 InfiniBand networking. It highlights the performance benefits for AI training and inference, noting up to 2x faster performance than Hopper GPUs and up to 15x faster inference on specific models. The post also emphasizes CoreWeave's infrastructure optimizations, including SUNK for unified training/serving and CKS for bare-metal performance, along with observability features.
  33. CoreWeave Launches the First Generally Available NVIDIA RTX PRO 6000 Blackwell Server InstancesCoreWeave launches generally available NVIDIA RTX PRO 6000 Blackwell Server Edition instances, marking the first cloud provider to offer this GPU architecture. The post details performance improvements over the L40S generation, including 5.6x faster LLM inference and 3.5x faster text-to-video generation, attributed to 96GB GDDR7 memory, fifth-generation Tensor Cores, and fourth-generation RT Cores. It highlights configurations of up to 8 GPUs per instance, paired with dual Intel Emerald Rapids CPUs and NVIDIA BlueField-3 DPUs for enhanced security and networking. The instances include over 7TB of NVMe storage and integrate with CoreWeave's Observability services, CKS, SUNK, CAIOS, and LOTA.
  34. CoreWeave Leads Artificial Analysis Kimi K2.6 Benchmark | CoreWeave BlogThis post details CoreWeave's leading performance and price for Kimi K2.6 inference, attributing it to NVFP4 quantization and EAGLE3 speculative decoding. It also highlights the use of NVIDIA GB300 and GB200 NVL72 clusters, NVIDIA Model Optimizer PTQ, and rigorous evaluation using Weights & Biases and modern test harness frameworks. The post emphasizes the integration of bare-metal access, high-speed interconnects, and custom inference stack tuning within the CoreWeave Inference stack.
  35. CoreWeave Leads the Way with First NVIDIA GB300 NVL72 DeploymentThis post announces the first deployment of the NVIDIA GB300 NVL72 platform on CoreWeave's cloud. It details the hardware specifications of the GB300 NVL72, including Blackwell Ultra GPUs, HBM3e memory, NVLink, and InfiniBand networking, and highlights its integration into CoreWeave's existing infrastructure (CKS, SUNK, observability tools, RLCC). The post emphasizes the performance gains for AI reasoning workloads and the collaborative effort with partners Dell Technologies, Switch, and Vertiv.
  36. CoreWeave Sets AI Benchmark Records with MLPerf v5.0 ResultsThis post details CoreWeave's achievement of new industry records in MLPerf Training v5.0 using the largest NVIDIA Blackwell GPU cluster (2,496 Blackwell GPUs). It highlights a 2x faster training time for Llama 3.1 405B model compared to Hopper GPUs and emphasizes the purpose-built AI infrastructure (AI-optimized data centers, bare-metal access, high-performance storage, NVIDIA Quantum InfiniBand and BlueField DPUs, SUNK, CKS, Mission Control) that drives this performance. The post quantifies customer benefits such as up to 2x faster training, 20% higher performance, and 14% better price-adjusted performance.
  37. CoreWeave to Offer NVIDIA HGX H100 Supercomputers | CoreWeaveThis post announces the availability of NVIDIA HGX H100 supercomputers on CoreWeave's platform. It highlights the performance improvements of the H100 over the A100 for HPC, AI training, and inference, and mentions the integration with NVIDIA Quantum-2 InfiniBand for low latency. The offering is framed within CoreWeave's existing Kubernetes-native infrastructure, emphasizing enterprise-grade features, flexible storage, and high-performance networking. The pricing model is reiterated as a competitive advantage.
  38. CoreWeave, NVIDIA, and IBM Set MLPerf Record with Largest NVIDIA GB200 Blackwell Cluster, Achieving Over 2× Faster TrainingThis post details CoreWeave's record-breaking MLPerf Training v5.0 submission using 2,496 NVIDIA Blackwell GPUs, achieving over 2x faster training performance than NVIDIA Hopper-based systems. It highlights the largest NVIDIA Blackwell cluster submission to date and the fastest training results on the Llama 3.1 405B benchmark. The post quantifies infrastructure density improvements with GB200 NVL72, showing a 4x reduction in physical space. It also references previous MLPerf records for inference and training, emphasizing CoreWeave's AI-optimized infrastructure and its ability to deliver scale, speed, and efficiency for AI workloads through its CKS and SUNK solutions.
  39. Decrease PyTorch Model Load Times with CoreWeave’s Tensorizer | CoreWeaveThis post introduces CoreWeave Tensorizer, a tool designed to drastically reduce PyTorch model loading times. It achieves this through a bespoke serialization format that enables 'zero-copy' tensor streaming directly from S3/HTTP endpoints or local storage to the GPU. Key technical contributions include: a novel serialization format placing metadata at the beginning of a single binary file for quick loading; 'tensor streaming' that pulls model chunks over the network wire speed to the GPU, minimizing RAM usage; and a comparison demonstrating >5x faster average latency per request compared to Hugging Face when scaling from zero, with fewer pod spin-ups and less RAM required. Tensorizer also offers sharding capabilities via a filter function and HTTP range support, and mitigates security risks associated with PyTorch's default pickle format by using a single binary file.
  40. Deploying NVIDIA H200 Tensor Core GPUs | CoreWeaveThis post announces the first deployment of NVIDIA H200 Tensor Core GPUs by CoreWeave, detailing their technical specifications (4.8 TB/s memory bandwidth, 141 GB HBM3e memory) and performance advantages over H100s for generative AI workloads. It outlines the instance configuration, including Intel fifth-generation Xeon CPUs, NVIDIA BlueField-3 DPUs, and 3200Gbps NVIDIA Quantum-2 InfiniBand networking, and mentions cluster sizes up to 42,000 GPUs. The post also reiterates the role of CoreWeave Mission Control for cluster reliability and resiliency, the benefits of CKS for bare-metal performance, and the integrated SUNK for job scheduling. Observability services are presented as a key component for identifying and resolving system faults.
  41. GA NVIDIA GB200 NVL72 Instances | CoreWeaveCoreWeave announces the general availability of NVIDIA GB200 NVL72 instances, making them the first cloud provider to offer this hardware. The post details the integration of these instances into CoreWeave's platform, including enhancements to CoreWeave Kubernetes Service (CKS) and Slurm on Kubernetes (SUNK) for optimized scheduling across NVLink and rack topologies. It highlights the use of NVIDIA Quantum-2 InfiniBand for 400Gb/s bandwidth per GPU and SHARP for offloaded collective communication, as well as NVIDIA BlueField-3 DPUs for multi-tenant networking and accelerated data access. The Observability Platform is updated with custom dashboards for GB200 NVL72 performance metrics like NVLink performance, GPU utilization, and temperature.
  42. How Autoscaling Impacts Compute Costs for Inference | CoreWeaveThis post analyzes the impact of autoscaling on compute costs for inference workloads. It details how faster model loading times and pod spin-up times, coupled with efficient autoscaling strategies, can significantly reduce inference costs without compromising performance. The post presents cost analysis scenarios comparing no autoscaling, slow autoscaling, and fast autoscaling, highlighting the benefits of responsive autoscaling for optimizing compute usage and improving user experience. It also discusses factors influencing autoscaling quality, such as GPU type, networking, and virtualization layers, and contrasts CoreWeave's bare-metal approach with generalized cloud providers.
  43. How the MosaicML Platform Makes Large-Scale Model Training More AccessibleThis post details MosaicML's integration with CoreWeave Cloud to provide access to NVIDIA H100 Tensor Core GPUs. It highlights performance benchmarks showing H100s with FP8 are 3x faster and ~30% more cost-effective than A100s with BF16 for the MPT-7B model. The post also mentions the use of NVIDIA InfiniBand networking (3.2 Tb/sec bandwidth) in CoreWeave's H100 instances and MosaicML's integration with the NVIDIA Transformer Engine library for system optimization.
  44. How to Serve Inference Faster with Infrastructure That Scales Securely | CoreWeaveThis post details the challenges of serving AI inference in the cloud, including high latency, GPU access bottlenecks, high infrastructure costs, limited interoperability, and scaling difficulties with traditional cloud infrastructure. It highlights how the generative AI boom exacerbates these issues due to ballooning model sizes. The post introduces CoreWeave Inference Service as a solution designed to address these challenges by offering high performance (throughput and accuracy), great user experiences (low latency, near real-time inference), and cost-effectiveness through optimized hardware usage on a bare-metal infrastructure. It implicitly contrasts this with legacy cloud provider approaches that rely on hypervisors and older designs.
  45. Inference Benchmarks: CoreWeave’s Inference Solution 8 to 10x Faster Than a Popular Generalized CloudThis post provides detailed benchmarks comparing CoreWeave's inference solution against a popular generalized cloud provider. It quantifies performance improvements in container spin-up time (3-5x faster), object storage transfer speed (3-5x faster, up to 12GB/s), and overall ML inference latency (8-10x faster) using EleutherAI's GPT-J-6B. The technical delta lies in the direct comparison of bare-metal infrastructure versus hypervisor-based generalized cloud offerings, explaining how CoreWeave's serverless Kubernetes approach and accelerated object storage contribute to these performance gains.
  46. Looking Ahead to 2023: How CoreWeave Is Using The Best NVIDIA GPUs | CoreWeaveThis post details the upcoming availability of NVIDIA H100 Tensor Core GPUs on CoreWeave Cloud in Q1 2023, highlighting their performance benefits for AI and ML workloads, specifically mentioning up to 9x faster training and 30x inference speedup for LLMs. It also mentions past integrations with EleutherAI for GPT-NeoX-20B training, and partnerships with Determined.AI and Zeet. The post also briefly mentions object storage and the CoreWeave Accelerator Program. A new NYC data center (LGA1) is noted for its low latency and storage capabilities.
  47. MLPerf Results: A Record-Breaking, Cloud-Native AI Supercomputer | CoreWeaveCoreWeave and NVIDIA achieved record-breaking performance on MLPerf Training 3.0, specifically for the GPT-3 175B LLM benchmark, training in under 11 minutes on over 3,500 NVIDIA H100 Tensor Core GPUs. This was accomplished on a production cluster featuring NVIDIA Quantum-2 InfiniBand networking, HGX H100 instances, Intel 4th Generation Xeon Scalable Processors, and NVIDIA ConnectX-7 400Gb/s InfiniBand and BlueField-2 DPUs. The post highlights the scale (over 3,500 GPUs, 4x larger than the next competitor) and speed (29x faster than the next best competitor) achieved, emphasizing CoreWeave's specialized infrastructure for AI workloads and its role as a 'utility provider' for AI compute.
  48. NLP-as-a-Service – The Next Evolution in AIThis post announces GooseAI, a fully managed NLP inference service built on CoreWeave Cloud, offering feature parity with existing APIs at a lower cost (50% savings mentioned, with a claim of over 70% cost savings by changing one line of code). It highlights the availability of open-source models like GPT-NeoX-20B and emphasizes CoreWeave's advantages such as fast spin-up times and responsive auto-scaling for serving these models. The post also references previous investments in AI infrastructure, including an A100 cluster for distributed training and partnership with EleutherAI for GPT-NeoX-20B.
  49. NovelAI as Among the First to Have NVIDIA HGX H100 GPUs Online | CoreWeaveThis post announces the general availability of NVIDIA HGX H100 GPUs on CoreWeave, highlighting NovelAI as an early adopter. It details CoreWeave's infrastructure for these GPUs, including a rail-optimized design, NVIDIA Quantum-2 InfiniBand networking (3.2Tbps per node), and the ability to scale to 16,384 H100 SXM5 GPUs. It also mentions the integration of NVIDIA BlueField DPUs for secure provisioning. The post emphasizes the benefits for large language model training and inference, enabling flexibility in model design, faster iteration, and serving models to millions of users.
  50. NVIDIA Blackwell Clusters on CoreWeave | CoreWeaveThis post announces the availability of NVIDIA Blackwell clusters (HGX B200 and GB200 NVL72) on CoreWeave, interconnected with NVIDIA Quantum-2 InfiniBand. It highlights the performance gains for AI inference, particularly for trillion-parameter LLMs, and mentions potential applications in scientific computing. The post also reiterates CoreWeave's focus on fully managed infrastructure for AI/ML customers, including tools for node health-checking and fast model loading.
  51. NVIDIA H100 Benchmarks for Large-Scale Training | CoreWeaveThis post details a benchmarking study on NVIDIA H100 GPUs for large-scale training, focusing on reliability and performance. It highlights the impact of infrastructure architecture, including bare-metal hardware (dual-fabric InfiniBand/Ethernet), topology-aware scheduling (SUNK), and optimized storage/checkpointing (Tensorizer, custom tokenizer), on achieving higher MTTF (3.66 days at 1024 GPUs, a 10x improvement) and MFU (51-52% vs. 35-45% typical). The study also quantifies the business impact of these improvements, showing faster model delivery and reduced iteration time.
  52. Pretraining vs. Fine-Tuning vs. RAG: Choosing the Right AI ApproachThis post introduces and compares three primary approaches to building AI models: pretraining, fine-tuning, and Retrieval-Augmented Generation (RAG). It details the pros, cons, costs, and time investments associated with each approach, offering guidance on selecting the most suitable method for different project needs. The post also provides real-world examples and discusses hybrid strategies, including the use of techniques like LoRA, QLoRA, multi-hop retrieval, and query decomposition.
  53. Pushing Boundaries with NVIDIA GB200 Clusters | CoreWeaveThis post announces the bring-up of one of the industry's first NVIDIA GB200 NVL72 clusters, detailing its technical specifications including NVIDIA NVLink, NVLink switch trays, and NVIDIA Quantum-2 InfiniBand networking, delivering up to 1.4 exaFLOPS of AI compute power per rack. It also introduces generally available NVIDIA GH200 Grace Hopper Superchip instances, highlighting their integrated CPU-GPU design, HBM3 memory, and CPU memory, along with 100 Gbps Ethernet and 7TB NVMe storage. New L40 and L40S GPU instances are also detailed, featuring 48GB GDDR6 memory, Tensor Cores, and FP8 support for L40S. A preview of CoreWeave AI Object Storage with LOTA™ technology is announced, promising up to 2 GB/s per GPU throughput. The post also mentions integration with CoreWeave Kubernetes Service (CKS) and Slurm on Kubernetes (SUNK).
  54. Right-Sizing AI Training with NVIDIA A100 & A40 on CoreWeaveThis post provides a comparative analysis of NVIDIA A100 and A40 GPUs for AI training workloads. It details performance benchmarks, specifically mentioning a 20B parameter model training time on A100s versus A40s, and estimates a 30% cost saving by right-sizing to A40s. It also highlights the technical specifications of both GPUs, including CUDA cores, Tensor Cores, memory, and bandwidth, and discusses CoreWeave's large inventory of A40 GPUs and their NVLink capabilities for increased interconnect bandwidth. The post also touches on A100's ability to be partitioned into seven GPU instances and its unified memory for large models.
  55. Serving LLM Inference with NVIDIA Triton and Eleuther AIThis post details the use of NVIDIA Triton Inference Server with the FasterTransformer backend to serve LLM inference for Eleuther AI's GPT-J and GPT-NeoX-20B models. It highlights performance improvements, specifically up to a 40% reduction in latency compared to HuggingFace. The post also emphasizes CoreWeave Inference Service's fast instance spin-up times (10 seconds for GPT-J, 15 seconds for NeoX) and responsive auto-scaling capabilities for LLM inference workloads.
  56. Support for NVIDIA AI Enterprise Software Platform and NVIDIA Cloud Functions | CoreWeaveThis post announces and details the integration of NVIDIA AI Enterprise software platform and NVIDIA Cloud Functions (NVCF) on the CoreWeave Cloud Platform, specifically for NVIDIA Hopper GPUs and NVIDIA GB200 NVL72 instances. It explains how NVCF, a core technology of DGX Cloud Serverless Inference, enables auto-scaling AI inference solutions and simplifies hosting AI inference workloads like NVIDIA NIM microservices. The post highlights the use of CoreWeave Kubernetes Service (CKS) with its optimized components for this integration and provides instructions on how customers can enable their CKS clusters to act as deployment targets for NVCF.
  57. Tarteel Migrates to CoreWeave with Support from ZeetThis post details Tarteel's migration of its AI-powered Quran app infrastructure from AWS to CoreWeave, facilitated by Zeet. The migration resulted in a 22% latency improvement and a ~56% cost reduction. Tarteel leveraged NVIDIA Riva and NeMo toolkits on CoreWeave to train and fine-tune ASR and TTS models, achieving optimized inference speeds with NVIDIA Triton. The post highlights CoreWeave's advantages in GPU selection, observability (Grafana/Prometheus), S3 integration, and Zeet's role in simplifying deployment and autoscaling, enabling Tarteel to focus on product development.
  58. The Power of the NVIDIA GB200 NVL72 | CoreWeaveThis post details the live demonstration and performance evaluation of the NVIDIA GB200 NVL72 on CoreWeave's platform. It showcases NCCL Allreduce and GPUBlaze tests to highlight the interconnectivity and computational power of the 72 GPUs. The post also describes a Megatron training run using CoreWeave's CKS and SUNK, demonstrating distributed workload efficiency and the use of CoreWeave Observability Services for monitoring GPU utilization, memory, power consumption, and temperature. It further mentions the integration with NVIDIA Quantum-2 InfiniBand networking and its SHARP In-Network Computing technology.
  59. Red Hat AI Inference on CKS | CoreWeave BlogIntroduces a deployment blueprint for running Red Hat AI Inference on CoreWeave Kubernetes Service (CKS) to enable hybrid inference across on-premises and cloud environments. This reference architecture allows enterprises to run the same open-source inference stack on-premises and on CoreWeave, leveraging Kubernetes-native control and open runtimes. It complements CoreWeave's existing inference portfolio by offering a supported path for self-managed inference on CKS using Red Hat's stack, building on the collaboration around the llm-d project.
  60. CoreWeave Launches NVIDIA GB200 Grace Blackwell Systems at ScaleThis post announces the launch of NVIDIA GB200 NVL72 systems at scale on CoreWeave, highlighting their availability to key customers like IBM, Cohere, and Mistral AI. It emphasizes CoreWeave's speed in operationalizing advanced AI systems and mentions achieving a new industry record in AI inference with these chips in MLPerf v5.0. The post also references previous early adoption of NVIDIA H100 and H200 GPUs and a demo of GB200 NVL72 systems. It notes the integration of these systems with CoreWeave's existing services like Kubernetes Service and SUNK, and mentions scaling capabilities up to 110,000 Blackwell GPUs with InfiniBand networking.
  61. CoreWeave Achieves New Record-Breaking AI Inferencing Benchmark with NVIDIA GB200 Grace Blackwell SuperchipsCoreWeave submitted MLPerf v5.0 Inference Closed results for NVIDIA GB200 Superchips, achieving 800 TPS on Llama 3.1 405B, and for NVIDIA H200 GPU instances, achieving 33,000 TPS on Llama 2 70B (a 40% improvement over H100). This demonstrates CoreWeave's infrastructure performance for large-model inference.