The evaluation engine behind Decagon’s AI agents | Decagon

This post details Decagon's model evaluation engine for AI agents. It introduces a two-phased framework: offline evaluation using LLM-as-judge and ground truth benchmarks, and online A/B testing with real customers. The offline phase scores responses on relevance, correctness, naturalness, and empathy, with human audits for validation. The online phase tracks business-critical metrics like CSAT and resolution rate. The post emphasizes that evaluation extends beyond response generation to the entire agent architecture, including retrieval and reasoning, and highlights the continuous feedback loop for model and system improvement.