Speculative Decoding Acceleration

5 posts · Aug 2025 – Apr 2026

What this trail covers

Together AI introduces ATLAS (AdapTive-LeArning Speculator System), a novel runtime-learning accelerator for LLM inference. ATLAS dynamically improves speculative decoding performance by learning from historical patterns and live traffic, continuously aligning with the target model's behavior. This adaptive approach offers significant speedups compared to standard or custom speculators, outperforming specialized hardware. The system integrates with existing Together Turbo techniques and aims to reduce inference latency and costs.

All Together AI trails

Apr 2026Accelerate RL rollouts by up to 50% with distribution-aware speculative decodingThis post introduces Distribution-aware speculative decoding (DAS), a framework to accelerate Reinforcement Learning (RL) post-training rollouts. DAS addresses the rollout bottleneck by employing a training-free adaptive suffix tree drafter and a length-aware scheduling strategy. The suffix tree drafter leverages recent trajectories to adapt to evolving model weights without retraining, and its construction/cleanup is parallelized with minimal overhead. The length-aware scheduler balances load across GPUs and allocates speculation budgets within GPUs based on request length (Long, Medium, Short) to mitigate stragglers and reduce GPU idle time. Experimental results show up to 50% rollout time reduction in math RL and ~25% in code RL, with no degradation in reward quality.
Mar 2026AuroraThis post introduces Aurora, an open-source, RL-based framework for continuous, online adaptation of speculative decoding models. It addresses the problem of stale draft models in production by learning directly from live inference traces. Key contributions include a serve-to-train flywheel architecture, asynchronous RL formulation, Tree Attention for processing complex branching structures, and demonstrated adaptation to distribution shifts, achieving a 1.25x speedup over static speculators and outperforming offline training from scratch in mixed traffic scenarios.
Mar 2026Mamba-3Introduces Mamba-3, a new state space model (SSM) specifically optimized for inference efficiency. Key technical contributions include: a more expressive recurrence formula derived from an exponential-trapezoidal discretization scheme, complex-valued state tracking for richer dynamics, and a multi-input, multi-output (MIMO) variant that improves accuracy with minimal impact on decode latency. The architecture also incorporates QKNorm (BCNorm) for training stabilization and removes the short convolution by integrating convolution-like mechanisms within the SSM recurrence. Empirical results show Mamba-3 outperforming Mamba-2 and other linear alternatives on language modeling and retrieval tasks, with MIMO further boosting accuracy. The post also highlights the potential for hybrid models combining linear layers with self-attention for future LLM architectures.
Dec 2025Introducing AutoJudge: Streamlined inference acceleration via automated dataset curationIntroduced AutoJudge, a method for accelerating LLM inference via automated dataset curation for lossy speculative decoding. AutoJudge trains a classifier on transformer hidden states to identify "important" mismatches that affect downstream task quality, enabling the acceptance of more draft tokens per verification cycle. This results in 1.5-2x speedups over standard speculative decoding with minimal accuracy degradation, demonstrated on GSM8K and LiveCodeBench benchmarks. The method is compatible with existing inference frameworks and can be composed with techniques like EAGLE-2.
Dec 2025Together AI delivers fastest inference for the top open-source modelsThis post details significant advancements in Together AI's inference platform, focusing on achieving up to 2x faster serverless inference for open-source LLMs. Key technical contributions include: 1. Next-gen GPU hardware optimization for NVIDIA Blackwell architecture (GB200 NVL72), including FP8/FP4 compute and low-overhead scheduling. 2. Development of 'Together Kernels' for Blackwell, including FlashAttention-4 and fused MoE kernels. 3. Turbo optimization suite featuring near-lossless quantization to FP8/FP4 with architecture-aware calibration and block-wise scaling. 4. Production-grade speculative decoding with training-efficient algorithms, high-accuracy custom draft models, adaptive acceptance strategies, and fail-safe fallback mechanisms. 5. A scalable draft-model training pipeline supporting models up to 1T parameters, utilizing curriculum-based training, post-training recipes, data-mixing, and an alignment evaluation framework.
Advertisement
Oct 2025AdapTive-LeArning Speculator System (ATLAS): A New Paradigm in LLM Inference via Runtime-Learning AcceleratorsThis post introduces ATLAS, an adaptive-learning speculator system for LLM inference. Unlike static or custom speculators, ATLAS evolves at runtime by learning from usage patterns and live traffic. It builds upon the Together Turbo Speculator and achieves significant speedups (e.g., 2.65x faster for DeepSeek-V3.1) by dynamically optimizing the speculative decoding process. The post details the core concepts of speculative decoding, including acceptance rate (α) and draft speed (c), and presents an interactive calculator to visualize speedup factors based on these parameters. ATLAS's adaptive nature ensures continuous performance improvement as workloads evolve.
Aug 2025How Together AI Uses AI Agents to Automate Complex Engineering Tasks: Lessons from Developing Efficient LLM Inference SystemsThis post details the application of AI agents to automate complex engineering tasks, specifically focusing on accelerating LLM inference via speculative decoding. It outlines six patterns for building automation agents: Infrastructure Patterns (Good Tools, Documentation, Safe Execution) and Behavioral Patterns (Manage Parallel Sessions, Manage Wait Time, Progress Monitoring). The case study demonstrates how these patterns were applied to build an automated speculator training pipeline, reducing manual intervention and improving efficiency.