Speculation Is All You Need | Modal Blog

This post details Modal's commitment to speculative decoding for LLM inference, highlighting its significant speedup potential (2x-3x) compared to traditional kernel optimizations. It introduces new DFlash speculators for Qwen models, achieving over 1000 tok/s on Qwen 3.5 122B-A10B. The post explains the mechanics of speculative decoding, emphasizes the ease of training custom speculators due to abundant and easily manipulable data, and showcases Modal's infrastructure for accelerating this training process. It also provides mathematical models and SGLang mocking to understand the impact of acceptance length on performance.