Supercharging LLM inference on Google TPUs: Achieving 3X speedups with diffusion-style speculative decoding- Google Developers Blog

What this post added

This post details the successful implementation of DFlash, a diffusion-style speculative decoding method, on Google TPUs within the vLLM inference framework. Key technical contributions include the 'dual-cache' solution for attention compatibility with paged attention, intelligent context management using power-of-2 padding for efficient host-TPU communication, and bridging the metadata gap in TPU inference to ensure strict alignment between the draft and target models. Benchmarking demonstrated an average 3.13x speedup on TPU v5p, with peak gains of nearly 6x on math tasks, and a 2.29x end-to-end serving speedup compared to EAGLE-3. The 'K-Flat' breakthrough, showing minimal cost for verifying larger token blocks on TPUs, is also highlighted.