I'm sorry, but those are vanity evals

This post introduces a 'funnel' approach to LLM evaluation, decomposing binary pass/fail metrics into a series of cascading steps. It uses GPT-4.1 as a case study to demonstrate how this method provides actionable insights beyond simple pass rates. Key contributions include: 1. Identifying 'eval pass rate' as a vanity metric and advocating for 'clarity metrics'. 2. Explaining the 'funnel' methodology: decomposing tasks into masked, sequential steps (e.g., RAG retrieval, table selection, column matching, query execution, correct data return). 3. Analyzing GPT-4.1's performance through the funnel, revealing a behavioral change (lowercasing/quoting identifiers) that impacted query execution, and highlighting significant upstream improvements in table selection. 4. Discussing the challenges of real-world semantic accuracy in SQL generation evals compared to syntactic ones.