Introducing AutoJudge: Streamlined inference acceleration via automated dataset curation

What this post added

Introduced AutoJudge, a novel method for accelerating LLM inference via task-specific lossy speculative decoding. AutoJudge automates the identification of "important" mismatches by re-evaluating tasks with swapped tokens, trains a lightweight classifier on existing embeddings to distinguish critical from non-critical mismatches, and accepts unimportant mismatches during verification to increase accepted tokens per cycle. Achieved 1.2-2.4x speedups on mathematical reasoning and programming tasks with minimal accuracy drops, and demonstrated compatibility with vLLM, TensorRT-LLM, and TGI, as well as composability with EAGLE-2.