Achieve state-of-the-art inference latencies with speculative decoding | Modal Blog

This post details how Modal applies speculative decoding to achieve state-of-the-art inference latencies. It elaborates on the low-latency playbook, focusing on reducing communication latency (using Modal Servers), host overhead (optimizing inference engines like SGLang), and prefill latency (using high-quality kernels). The core contribution is the application and optimization of speculative decoding, including the use of custom speculator models like DFlash, fine-tuned on task-specific synthetic data, to significantly improve acceptance lengths and overall inference throughput. It highlights the integration with Modal Servers for low-latency regional deployment and the use of optimized kernels and inference engines.