Speculative Decoding for LLM Inference

7/27/2026

This post details the integration of Moonshot's Kimi K3 model with Modal's platform, specifically highlighting the application of a custom-trained DFlash speculator tuned to K3's architecture. This integration resulted in a 360% faster interactivity (from 100 to 460 tokens per second) and 88% higher throughput (from 800k to 1.5 million TPM per GPU) on agentic workloads. The post also touches upon the engineering challenges and solutions involved in making Kimi K3, a 2.8 trillion parameter multimodal model with a 1M token context window, servable at scale, including its mixture-of-experts architecture, quantization-aware training, and contributions to vLLM for prefix caching.

7/15/2026

This post details the integration of the Inkling multimodal model on Modal, highlighting its architecture (mixture-of-experts, local attention) and the performance gains achieved through a custom DFlash speculator tuned for this model shape. It specifically details how DFlash was adapted with all-local attention and causal layers for better kernel support to achieve 250 tokens per second per user on agentic workloads. It also announces Inkling's availability as a Managed Endpoint.

7/1/2026

Introduced Multi-Token Residual Prediction (MRP) for diffusion language models (DLMs). MRP trains a small module to predict the residual between adjacent denoising steps, rather than the full next-step distribution. This enables DLMs to achieve speedups in static denoising regimes (e.g., speculative decoding with up to 1.56x throughput) and recover quality lost in dynamic denoising regimes (up to +16 accuracy points). The post details the challenges of direct MTP in DLMs and the architectural changes for MRP, including a frozen DLM backbone and a trained MRP module. It also outlines applications in speculative and direct decoding, with performance metrics on various benchmarks.

6/24/2026

This post details the application of speculative decoding for achieving state-of-the-art inference latencies, specifically highlighting its use with Modal Auto Endpoints and Modal Servers. It elaborates on the low-latency playbook, emphasizing optimizations in client-server communication, host overhead, prefill latency, and decode latency. A significant contribution is the detailed explanation of applying speculative decoding with custom speculator models, including 'mid-training' on task-specific synthetic data using techniques like DFlash. The post also discusses the integration with open-source engines (SGLang) and kernels (FlashAttention-4) and provides a case study with Decagon Voice, demonstrating a 100ms reduction in p50 latency.

6/19/2026

This post introduces and advocates for speculative decoding as a primary engine optimization for LLM inference. It details the mechanism of speculative decoding, explaining how a draft model generates token sequences that are then validated by the target model, leading to lossless acceleration. The post highlights the importance of acceptance length and demonstrates how training custom speculators, particularly using architectures like DFlash, can yield significant speedups (2x-3x). It also emphasizes that speculator training is a simplified ML problem due to readily available data and a clear objective metric, and showcases how Modal's infrastructure accelerates this training process, enabling application-specific optimizations.

11/13/2025

This post details Decagon's collaboration with Modal to improve speculative decoding for LLM inference in real-time voice AI. Key contributions include: 1. Forking and building a custom training pipeline for EAGLE3 draft models using SpecForge, pre-training on broad conversational data and fine-tuning on Decagon's domain dataset, resulting in 38% higher accept lengths. 2. Re-engineering the SGLang inference engine by building a new asynchronous scheduler for speculative decoding, eliminating CPU stalls and enabling parallel GPU kernel execution, which improved throughput by up to 12% and achieved p90 latency of 342ms.