A quick introduction to speculative decoding

This post introduces speculative decoding as a technique to reduce LLM inference latency. It explains the core concept of using a smaller 'draft' model to generate candidate tokens that a larger 'target' model validates, thereby increasing the number of tokens generated per forward pass. The post details the step-by-step process, discusses limitations such as reduced throughput with larger batch sizes and increased per-token costs, and outlines factors for a good speculative decoding setup (draft model selection, fine-tuning, resource allocation). It highlights use cases like code generation where latency is critical and batch sizes are naturally small. The post also points to Baseten's integration of speculative decoding into its TensorRT-LLM Engine builder.