DFlash: 3x faster LLM inference

This post details Baseten's implementation of DFlash, a novel speculative decoding technique for LLM inference. It explains how DFlash achieves up to 3x speedups by predicting multiple tokens in parallel using block diffusion and bidirectional attention, surpassing autoregressive methods like EAGLE and improving upon vLLM's DFlash implementation. The post covers Baseten's custom forward pass mechanism for preparing the KV cache and producing output tokens, guided decoding support, and presents benchmark results across GSM8k, MATH-500, and NVIDIA's Nemotron datasets. It also outlines the process for training DFlash draft models, including input requirements, loss weighting, and attention mask structure.