Hardware-Aware, Dynamic Speculative Decoding (DSD) | Cohere

What this post added

This post introduces Hardware-Aware, Dynamic Speculative Decoding (DSD) for LLM inference. DSD dynamically adjusts the number of draft tokens (K) based on hardware characteristics (memory bandwidth vs. compute-bound regimes) and model type (dense vs. MoE). It uses 'goodput' (Acceptance Length / Inter-Token Latency) as a metric to find the optimal K, which is stored in a lookup table. The post details the challenges of speculative decoding in production, the non-monotonic behavior of optimal K for MoE models, and the integration of DSD into vLLM, highlighting its compatibility with async scheduling and CUDA Graphs. Benchmarks show DSD outperforming fixed-K speculative decoding and vanilla inference at higher batch sizes for dense models.