Teaching Sidekick to say no: automated data curation with LLM judge consensus (2026) - Shopify

This post details the development of an automated data curation pipeline for training LLMs to refuse impossible queries. It introduces the use of an LLM judge ensemble (Frontier LLM A, B, C, and D) calibrated with a small, high-quality annotated dataset to evaluate production queries. The system requires strict consensus among judges for label acceptance, prioritizing precision over recall. The pipeline categorizes queries into 'solvable with more context', 'missing capability', 'wrong skill', and 'ambiguous'. The outcome shows a significant improvement in segmentation skill evaluation scores from 0.619 to 0.798, with refusal accuracy at 86.3% and a false positive rate of 4.6%.