DuetBench-2: Measuring agent self-improvement in production | Decagon

Introduced DuetBench-2, a more rigorous benchmark for evaluating AI agent self-improvement, focusing on verifiable capability, production impact, and compounding self-improvement. Detailed Autopilot's performance against this new benchmark, achieving an 81.9% pass rate for AOP building/revision, with specific metrics on build time and human intervention. Highlighted improvements in simulation generation (96% pass rate) and LLM-as-a-judge accuracy (99.5%). Quantified production impact with a 25pp resolution rate increase at a Fortune 50 company and demonstrated durability with over 85% of updates remaining live after 30 days. Provided an example of compounding self-improvement through a self-repaired polling loop in the simulation runner.