A statistical approach to model evaluations

This post introduces a statistical approach to AI model evaluations, advocating for the reporting of standard error of the mean (SEM) and clustered standard errors to quantify uncertainty and improve the rigor of comparisons between models. It recommends reporting pairwise information (mean differences, standard errors, confidence intervals, and correlations) and applying power analysis to determine appropriate eval sizes. The core technical contributions include the application of statistical theory (Central Limit Theorem, clustered standard errors, two-sample t-tests, paired-differences tests) to the problem of AI model evaluation, aiming to distinguish real capability differences from random chance.