Blogs›Apple›Speculative Decoding for LLM Inference

Speculative Decoding for LLM Inference

posts

2026

This post introduces ARBITRAGE, a novel step-level speculative generation framework for Large Language Models (LLMs). ARBITRAGE dynamically routes generation based on the predicted advantage between draft and target models, using a lightweight router to determine when the target model is likely to produce a meaningfully better step. This approach approximates an ideal ORACLE, achieving near-optimal efficiency-accuracy trade-offs and reducing inference latency by up to 2x at matched accuracy on m. This work extends prior research by scaling Categorical Flow Maps (CFMs) to 1.7B parameters and 2.1T tokens, demonstrating competitive sample quality and near-data-level token entropy in as few as 4 inference steps. It also introduces a likelihood bound for CFMs in the semi-discrete setting and provides insights into challenges and solutions for training these models at scale, including loss weighting and time scheduling.

2026

Arbitrage: Efficient Reasoning via Advantage-Aware Speculation

8/7/2026

Introduces ARBITRAGE, a step-level speculative generation framework that uses an advantage-aware router to dynamically decide whether to accept a step from a draft model or regenerate with a target model. This differs from prior methods by not relying on fixed acceptance thresholds and by training a lightweight router to predict the relative advantage of the target model, thereby reducing unnecessary rejections and wasted target compute.

Scaling Categorical Flow Maps

8/7/2026

This post scales Categorical Flow Maps (CFMs) to a 1.7B-parameter model trained on 2.1T tokens, demonstrating competitive sample quality and near-data-level token entropy in as few as 4 inference steps. It introduces a likelihood bound for CFMs in the semi-discrete setting for scoring on LM benchmarks and provides prescriptive insights on loss weighting and time scheduling for training at scale.