Speculative Decoding Acceleration

3/31/2026

Introduces Aurora, an open-source, RL-based framework for continuously updating speculative decoding models from live inference traces. This system addresses the issue of stale draft models in production by enabling real-time adaptation to shifting traffic domains and outperforming static speculators. Key technical contributions include a serve-to-train flywheel powered by RL, decoupled inference and training servers, lazy and non-disruptive synchronization, a Tree Attention mechanism for processing speculative decoding results, and adaptation to distribution shift through continuous learning.

3/17/2026

Introduces Mamba-3, a new state space model (SSM) specifically optimized for inference efficiency. Key technical contributions include: a more expressive recurrence formula derived from an exponential-trapezoidal discretization scheme, complex-valued state tracking for richer dynamics, and a multi-input, multi-output (MIMO) variant that improves accuracy with minimal impact on decode latency. The architecture also incorporates QKNorm (BCNorm) for training stabilization and removes the short convolution by integrating convolution-like mechanisms within the SSM recurrence. Empirical results show Mamba-3 outperforming Mamba-2 and other linear alternatives on language modeling and retrieval tasks, with MIMO further boosting accuracy. The post also highlights the potential for hybrid models combining linear layers with self-attention for future LLM architectures.

12/3/2025

Introduced AutoJudge, a novel method for accelerating LLM inference via task-specific lossy speculative decoding. AutoJudge automates the identification of "important" mismatches by re-evaluating tasks with swapped tokens, trains a lightweight classifier on existing embeddings to distinguish critical from non-critical mismatches, and accepts unimportant mismatches during verification to increase accepted tokens per cycle. Achieved 1.2-2.4x speedups on mathematical reasoning and programming tasks with minimal accuracy drops, and demonstrated compatibility with vLLM, TensorRT-LLM, and TGI, as well as composability with EAGLE-2.

12/1/2025

This post details significant advancements in Together AI's inference platform, focusing on achieving up to 2x faster serverless inference for open-source LLMs. Key technical contributions include: 1. Next-gen GPU hardware optimization for NVIDIA Blackwell architecture (GB200 NVL72), including FP8/FP4 compute and low-overhead scheduling. 2. Development of 'Together Kernels' for Blackwell, including FlashAttention-4 and fused MoE kernels. 3. Turbo optimization suite featuring near-lossless quantization to FP8/FP4 with architecture-aware calibration and block-wise scaling. 4. Production-grade speculative decoding with training-efficient algorithms, high-accuracy custom draft models, adaptive acceptance strategies, and fail-safe fallback mechanisms. 5. A scalable draft-model training pipeline supporting models up to 1T parameters, utilizing curriculum-based training, post-training recipes, data-mixing, and an alignment evaluation framework.

10/10/2025

Introduced ATLAS (AdapTive-LeArning Speculator System), a novel speculative decoding system that dynamically improves at runtime by learning from historical patterns and live traffic. ATLAS offers automatic performance improvements without manual tuning, continuously aligning with the target model's behaviors in real time. It achieves up to 2.65x speedup on DeepSeek-V3.1 and Kimi-K2, outperforming standard decoding and specialized hardware like Groq.