Together AI Delivers Top Speeds for DeepSeek-R1-0528 Inference on NVIDIA Blackwell

This post details the significant performance improvements for DeepSeek-R1-0528 inference on NVIDIA Blackwell GPUs, achieved through a combination of bespoke GPU kernels, a proprietary inference engine, speculative decoding methods (Together Turbo Speculator), and calibrated/quantized model optimization. It highlights the fastest serverless inference performance for DeepSeek-R1-0528 to date, with up to 334 tokens/sec on HGX B200. Dedicated Endpoints are also discussed as a further optimization layer for production workloads, offering up to an 84 tokens/sec speedup. The post elaborates on the technical components: the inference engine's use of FlashAttention-3 and CUDA graphs, the development of Blackwell GPU kernels leveraging 5th-gen Tensor Cores, the Turbo Speculator's improved acceptance rate over other methods, and a lossless quantization technique (NVFP4/MXFP4/6/8) preserving model accuracy.