Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference | NVIDIA Technical Blog

This post introduces speculative decoding as a technique to accelerate LLM inference by using a draft model to propose multiple tokens for parallel verification by a larger target model. It provides five guidelines for selecting the optimal draft length and draft mechanism, balancing acceptance length, draft overhead, and training/deployment costs. It also introduces SPEED-Bench for benchmarking speculative decoding performance and provides training examples for draft mechanisms like EAGLE-3, DFlash, and DSpark.