DuetBench: An evaluation of self-improving customer service agents | Decagon

This post introduces DuetBench, a novel benchmark for evaluating the self-improvement capabilities of AI customer service agents. DuetBench addresses limitations of existing benchmarks by focusing on dynamic, iterative work, the ability to self-improve, and real-world complexity. It comprises diagnostic, build, simulation-generation, and end-to-end certification tasks. The post presents results showing Duet Autopilot outperforms average human performance in diagnostic tasks (93% vs. 83% pass rate) and timed enterprise agent-building tasks (45.5% vs. 23% human pass rate within 3 hours), particularly for medium and hard complexity tasks. Autopilot also demonstrates strong self-critique capabilities, improving simulation accuracy from 58% to 88% and reaching the human certification standard (90% pass rate) for end-to-end agent building.