How to Evaluate AI Agents From Tool Calls to Task Completion | NVIDIA Technical Blog

This post details a new framework for evaluating AI agents, shifting focus from individual tool call accuracy to full task completion in stateful environments. It introduces a two-layer scoring system (step-level process scoring and end-to-end outcome scoring) and a hierarchical metric structure (Benchmark → Trial → Task → Turn → Step) with axes for accuracy, verbosity, and cost. Key metrics include task success rate, consistency, tool-call precision, argument accuracy, steps per success, and cost per success. The post emphasizes the importance of benchmark comparability through task complexity, environment statefulness, and executable verification methodologies, while also highlighting the risks of data contamination. The Nemotron 3.5 Lightning model is presented as an example demonstrating this evaluation framework.