CoreWeave Leads MLPerf 0.7 Endpoints | CoreWeave Blog

CoreWeave achieved leading results in the MLPerf 0.7 Endpoints benchmark by running the DeepSeek-R1 model on 68 NVIDIA Blackwell GPUs in a GB200 NVL72 system. The post details sustained system throughput of 441,740 output tokens per second at 16,384 concurrent requests and up to 6,496 tokens per second per GPU. It emphasizes that per-GPU throughput and time per output token (TPOT) are critical metrics for inference services, influencing both user experience and cost. The benchmark was run on standard production infrastructure, including bare-metal servers, NVIDIA Quantum-2 InfiniBand networking, CKS, SUNK for topology-aware scheduling, and DFS for high-throughput storage, demonstrating the platform's capability for real-world inference workloads.