Using AI Evaluator to ensure Glean Assistant meets modern enterprise needs

Introduced an LLM-based evaluation system (AI Evaluator) to automatically measure the performance of the Glean Assistant. This system evaluates both the retrieval (context relevance rate, context recall rate) and generation (answer relevance, answer completeness, groundedness) components. It addresses challenges like LLM biases and inconsistency by employing techniques such as single-answer grading, categorical rating, prompt engineering, and model ensembles. The system achieved 74% human agreement and improved user retention prediction accuracy by 18.8%.