Benchmarking AI Assistants

This post introduces a novel supervised metric, the 'Answer Alignment Score', for evaluating AI assistant answer quality. The metric involves extracting atomic facts from ground truth, using an LLM to classify generated answers against these facts (Entailed, Contradicts, Neutral), and aggregating these classifications into Correctness-Completeness scores. This approach aims to provide better alignment with human judgment than existing unsupervised metrics like RAGAS, reducing false positives from 0.94 to 0.027. The post also details the benchmark datasets used (FinanceBench, Open Australian Legal, NQ-HARD) and presents comparative results showing Pinecone Assistant outperforming OpenAI assistants.