TechPrufer
Get started
Apple
Apple logoApple · Feature trail

Diffusion vs. Autoregressive Language Model Performance

3 posts · Aug 2026 – Sep 2026

What this trail covers

This post introduces probe guidance, a method to guide flow matching models using frozen internal states of an existing diffusion model. This eliminates the need for an additional forward pass at inference time and ensures that weak and strong models share similar dynamics. Applied to continuous diffusion language models, probe guidance sets a new state-of-the-art performance on unconditional generation. When applied to a 1.7B diffusion language model, probe guidance consistently improves on multiple choice question answering benchmarks. The post also studies the autoguidance setting, finding that the weak model must come from a low-entropy region of training. This provides a practical way to improve diffusion language models and sheds light on the mechanism behind autoguidance.

All Apple trails
  1. How to Guide Your Language FlowIntroduces probe guidance, a novel method for guiding flow matching models by leveraging frozen internal states of an existing diffusion model. This technique avoids an extra inference pass and ensures dynamic similarity between weak and strong models. Benchmarked on continuous diffusion language models, it achieves state-of-the-art unconditional generation performance and improves multiple-choice question answering benchmarks when applied to a 1.7B parameter model. The work also investigates autoguidance, identifying the necessity of a low-entropy training region for the weak model.
  2. DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language ModelsDACA-GRPO introduces Denoising-Aware Credit Assignment for GRPO-style trainers in Diffusion Language Models. It addresses two weaknesses: lack of temporal credit assignment across denoising steps and bias in mean-field likelihood estimates. DACA-GRPO uses Denoising Progress Scores to extract per-token importance weights from intermediate predictions and Stratified Masking Likelihood to reduce mean-field bias by partitioning token positions. This plug-and-play enhancement achieves consistent improvements across mathematical reasoning, code generation, constraint satisfaction, and JSON schema adherence benchmarks.
  3. Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language ModelsThis post presents a comprehensive study of the performance characteristics of ARMs and DLMs, combining theoretical analysis with empirical profiling. It shows that DLMs can achieve higher arithmetic intensity than ARMs by leveraging parallelism across token positions but fail to scale effectively with longer contexts. It explores block-wise decoding for DLMs to enable better scaling to long contexts and examines batched inference, finding that ARMs exhibit superior throughput. Opportunities for accelerating DLM inference are highlighted, emphasizing that reducing the number of sampling steps is key for open-source DLMs to achieve lower latency relative to ARMs.