Fine-tuning open LLM judges to outperform GPT-5.2

This post details the successful fine-tuning of open-source LLM judges (GPT-OSS 120B, Qwen3 235B) using Direct Preference Optimization (DPO) to outperform GPT-5.2 in evaluating model outputs. The experiment demonstrated that fine-tuned open-source models can achieve superior performance at a fraction of the cost and with significantly faster inference speeds, as measured by agreement with human preference labels on the RewardBench 2 benchmark. The post provides a deep dive into the methodology, baseline evaluations, category-level analysis, and practical code for implementing this optimization.