TechPrufer
Sign inGet started

Blogs›Cohere›Dynamic Speculative Decoding

Dynamic Speculative Decoding

Dynamic Speculative Decoding

1
posts
2026

Cohere has developed and contributed Dynamic Speculative Decoding (DSD) to the vLLM inference framework. DSD adaptively controls the number of draft tokens based on hardware constraints (compute vs. memory bandwidth) and model architecture (dense vs. MoE), improving LLM inference speed and efficiency across various batch sizes and workloads, including large-scale RL rollouts. This optimization is compatible with vLLM's asynchronous scheduling and full CUDA Graph features.

2026

Hardware-Aware, Dynamic Speculative Decoding (DSD) | Cohere

7/10/2026

This post introduces Hardware-Aware, Dynamic Speculative Decoding (DSD) for LLM inference. DSD dynamically adjusts the number of draft tokens (K) based on hardware characteristics (memory bandwidth vs. compute-bound regimes) and model type (dense vs. MoE). It uses 'goodput' (Acceptance Length / Inter-Token Latency) as a metric to find the optimal K, which is stored in a lookup table. The post details the challenges of speculative decoding in production, the non-monotonic behavior of optimal K for MoE models, and the integration of DSD into vLLM, highlighting its compatibility with async scheduling and CUDA Graphs. Benchmarks show DSD outperforming fixed-K speculative decoding and vanilla inference at higher batch sizes for dense models.