How to Guide Your Language Flow

Introduces probe guidance, a novel method for guiding flow matching models by leveraging frozen internal states of an existing diffusion model. This technique avoids an extra inference pass and ensures dynamic similarity between weak and strong models. Benchmarked on continuous diffusion language models, it achieves state-of-the-art unconditional generation performance and improves multiple-choice question answering benchmarks when applied to a 1.7B parameter model. The work also investigates autoguidance, identifying the necessity of a low-entropy training region for the weak model.