Challenges in evaluating AI systems

This post details specific challenges encountered in developing and implementing AI evaluations. It highlights issues with multiple-choice evaluations like MMLU (training data contamination, sensitivity to formatting, inconsistent implementation, proofreading errors) and BBQ (implementation complexity, nuanced bias score interpretation, failure modes like models not answering). It also discusses the engineering overhead and scalability issues with third-party frameworks like BIG-bench and the prompting format inconsistencies with HELM. Finally, it outlines the cost, time, and variability challenges associated with using crowdworkers for A/B testing and red teaming, and the inherent tension between helpfulness and harmlessness in human evaluation.