Hill climbing to glory: using evals to improve AI error rate by 7x

This post details Hex's application of evals and a hill-climbing methodology to improve the error rate of its 'Quick Edits' AI feature by 7x. Key contributions include: prioritizing the reduction of wrong edits over incorrect handoffs, demonstrating that improvements to the validation harness (handling formatting, rounding, and property validation) were more impactful than prompt engineering, and using holdout cases (40% of the eval suite) to prevent overfitting. The post also highlights the importance of tuning the number of attempts per eval case (10 was found to be optimal) to reduce noise, and the benefit of fast eval runs (minutes, <$10) for rapid iteration on AI and feature development simultaneously.