Hardware-Aware, Dynamic Speculative Decoding (DSD) | Cohere

This post details the implementation and performance of hardware-aware Dynamic Speculative Decoding (DSD) within the vLLM inference framework. It introduces the concept of adapting the number of draft tokens (K) based on hardware constraints (compute vs. memory bandwidth) and model architecture (dense vs. MoE) to optimize LLM inference speed. The post quantifies the performance gains of DSD over vanilla inference and fixed-K speculative decoding across various batch sizes, highlighting its ability to maintain speedups even at high batch sizes where fixed-K SD can regress. It also discusses the critical compatibility challenges and solutions for integrating DSD with vLLM's asynchronous scheduling and full CUDA Graph features, providing concrete examples of how DSD adapts to changing workloads and model types.