Introducing AutoJudge: Streamlined inference acceleration via automated dataset curation

Introduces AutoJudge, a method to accelerate LLM inference via task-specific lossy speculative decoding. AutoJudge automates dataset curation for speculative decoding by training a classifier to identify "important" mismatches that affect downstream quality, removing the need for manual annotation. It achieves 1.5-2x speedups over standard speculative decoding by accepting more draft tokens per verification cycle with minimal accuracy drop. The method integrates with existing frameworks like vLLM and TensorRT-LLM and can be composed with other techniques like EAGLE-2.