DuetBench: An evaluation of self-improving customer service agents | Decagon

What this post added

Introduced DuetBench, a new evaluation framework for self-improving customer service agents. DuetBench measures the agent's ability to diagnose issues, build Agent Operating Procedures (AOPs), generate and validate test simulations, and perform end-to-end certification tasks. The framework mirrors real-world enterprise agent building with simulated customer deployments, AOPs, tools, guardrails, and synthetic traffic. Results show Duet Autopilot outperforms human baselines in diagnostic tasks (93% vs 83%), timed agent-building tasks (45.5% vs 23%), and simulation accuracy after self-critique (58% to 88%). Autopilot also achieved the human certification standard (90% pass rate). This demonstrates a shift towards agent managers overseeing automated agent-building work.