Speculative Decoding for LLM Inference

8 posts · Nov 2025 – Aug 2026

What this trail covers

Modal's platform deeply integrates speculative decoding for LLM inference, enabling significant speedups by using a draft model to generate token sequences in parallel. This feature allows for lossless acceleration of the autoregressive decoding phase, turning serial work into parallel work. The platform supports training custom speculators tailored to application-specific data, leading to substantial improvements in acceptance lengths and overall inference throughput. Recent work has focused on optimizing diffusi

All Modal trails

Aug 2026Qwen3.8-2.4T-A95B now available on Modal | Modal BlogThis post announces the availability of Qwen3.8-2.4T-A95B on Modal and highlights the use of a custom DFlash speculator tuned to Qwen3.8's shape to speed up inference. The speculator was trained with data that emphasizes coding, research, and work tasks to increase token acceptance rates, thereby improving inference speed.
Jul 2026Kimi K3 by Moonshot now available on Modal | Modal BlogThis post details the integration of Moonshot's Kimi K3 model with Modal's platform, specifically highlighting the application of a custom-trained DFlash speculator tuned to K3's architecture. This integration resulted in a 360% faster interactivity (from 100 to 460 tokens per second) and 88% higher throughput (from 800k to 1.5 million TPM per GPU) on agentic workloads. The post also touches upon the engineering challenges and solutions involved in making Kimi K3, a 2.8 trillion parameter multimodal model with a 1M token context window, servable at scale, including its mixture-of-experts architecture, quantization-aware training, and contributions to vLLM for prefix caching.
Jul 2026Inkling by Thinking Machines now available on Modal | Modal BlogThis post details the integration of the Inkling multimodal model on Modal, highlighting its architecture (mixture-of-experts, local attention) and the performance gains achieved through a custom DFlash speculator tuned for this model shape. It specifically details how DFlash was adapted with all-local attention and causal layers for better kernel support to achieve 250 tokens per second per user on agentic workloads. It also announces Inkling's availability as a Managed Endpoint.
Jul 2026Multi-token Residual Prediction | Modal BlogIntroduced Multi-Token Residual Prediction (MRP) for diffusion language models (DLMs). MRP trains a small module to predict the residual between adjacent denoising steps, rather than the full next-step distribution. This enables DLMs to achieve speedups in static denoising regimes (e.g., speculative decoding with up to 1.56x throughput) and recover quality lost in dynamic denoising regimes (up to +16 accuracy points). The post details the challenges of direct MTP in DLMs and the architectural changes for MRP, including a frozen DLM backbone and a trained MRP module. It also outlines applications in speculative and direct decoding, with performance metrics on various benchmarks.
Jun 2026Achieve state-of-the-art inference latencies with speculative decoding | Modal BlogThis post details how Modal applies speculative decoding to achieve state-of-the-art inference latencies. It elaborates on the low-latency playbook, focusing on reducing communication latency (using Modal Servers), host overhead (optimizing inference engines like SGLang), and prefill latency (using high-quality kernels). The core contribution is the application and optimization of speculative decoding, including the use of custom speculator models like DFlash, fine-tuned on task-specific synthetic data, to significantly improve acceptance lengths and overall inference throughput. It highlights the integration with Modal Servers for low-latency regional deployment and the use of optimized kernels and inference engines.
Advertisement
Jun 2026Speculation Is All You Need | Modal BlogThis post details Modal's commitment to speculative decoding for LLM inference, highlighting its significant speedup potential (2x-3x) compared to traditional kernel optimizations. It introduces new DFlash speculators for Qwen models, achieving over 1000 tok/s on Qwen 3.5 122B-A10B. The post explains the mechanics of speculative decoding, emphasizes the ease of training custom speculators due to abundant and easily manipulable data, and showcases Modal's infrastructure for accelerating this training process. It also provides mathematical models and SGLang mocking to understand the impact of acceptance length on performance.
Nov 2025How Decagon shipped real-time voice AI on ModalThis post details the collaboration between Decagon and Modal to achieve real-time voice AI inference. Key contributions include: 1. Training a custom EAGLE3 draft model using SpecForge and a specialized pipeline, resulting in a 38% higher accept length compared to open-source baselines. 2. Re-engineering the SGLang inference engine by building a new asynchronous scheduler for speculative decoding, eliminating host-side inefficiencies, enabling parallel GPU kernel execution, and achieving sync-free speculative decoding. This resulted in up to 12% throughput improvement and p90 latency of 342ms.