TechPrufer
Get started
Cohere
Cohere logoCohere · Feature trail

Dynamic Speculative Decoding

4 posts · Nov 2025 – Jul 2026

What this trail covers

Cohere has developed and contributed Dynamic Speculative Decoding (DSD) to the vLLM inference framework. DSD adaptively controls the number of draft tokens based on hardware constraints (compute vs. memory bandwidth) and model architecture (dense vs. MoE), improving LLM inference speed and efficiency across various batch sizes and workloads, including large-scale RL rollouts. This optimization is compatible with vLLM's asynchronous scheduling and full CUDA Graph features. This post further analyzes the performance gains and compatibility challenges of DSD within vLLM, demonstrating its effectiveness in production systems and RL rollouts.

All Cohere trails
  1. Production-Ready W4A8: vLLM Integration Explained | CohereThis post details the implementation of production-ready W4A8 GEMM kernels for NVIDIA Hopper GPUs, integrated into vLLM. It introduces a LUT-based solution for efficient INT4 to FP8 dequantization, a custom CUDA kernel for faster pre-processing, and a refined quantization recipe that uses token masking during calibration and compensates for FP8 range limitations to maintain model quality. The post quantifies performance improvements and accuracy preservation compared to naive FP8 casting and W4A16.
  2. Why MoE Models Get More From Speculative Decoding | CohereThis post validates and extends the prediction of a non-monotonic speedup curve for speculative decoding (SD) with Mixture-of-Experts (MoE) models. It analyzes the impact of temporal correlation in expert routing on SD verification cost, showing that expert overlap between adjacent tokens is significantly higher than predicted by independence baselines. The post also uncovers a distinct mechanism for high speedup at batch size 1, related to fixed-overhead amortization. Concrete design implications for co-optimizing model sparsity and speculative decoding are drawn.
  3. Hardware-Aware, Dynamic Speculative Decoding (DSD) | CohereThis post details the implementation and performance of hardware-aware Dynamic Speculative Decoding (DSD) within the vLLM inference framework. It introduces the concept of adapting the number of draft tokens (K) based on hardware constraints (compute vs. memory bandwidth) and model architecture (dense vs. MoE) to optimize LLM inference speed. The post quantifies the performance gains of DSD over vanilla inference and fixed-K speculative decoding across various batch sizes, highlighting its ability to maintain speedups even at high batch sizes where fixed-K SD can regress. It also discusses the critical compatibility challenges and solutions for integrating DSD with vLLM's asynchronous scheduling and full CUDA Graph features, providing concrete examples of how DSD adapts to changing workloads and model types.
  4. Making data transfer in LLM systems faster | CohereThis post introduces techniques to accelerate data transfer in LLM systems, focusing on optimizing the movement of data between different components of the LLM pipeline. It builds upon the existing Dynamic Speculative Decoding (DSD) capability by addressing bottlenecks in data loading, preprocessing, and inter-component communication, thereby further enhancing overall inference speed and efficiency.