AdapTive-LeArning Speculator System (ATLAS): A New Paradigm in LLM Inference via Runtime-Learning Accelerators

This post introduces ATLAS, an adaptive-learning speculator system for LLM inference. Unlike static or custom speculators, ATLAS evolves at runtime by learning from usage patterns and live traffic. It builds upon the Together Turbo Speculator and achieves significant speedups (e.g., 2.65x faster for DeepSeek-V3.1) by dynamically optimizing the speculative decoding process. The post details the core concepts of speculative decoding, including acceptance rate (α) and draft speed (c), and presents an interactive calculator to visualize speedup factors based on these parameters. ATLAS's adaptive nature ensures continuous performance improvement as workloads evolve.