Autopilot in production: Governance of self-improving agents | Decagon

This post details the introduction of Duet Autopilot, a self-improving agent for agent building, and its supporting governance platform. Key technical contributions include: 1. Vault: A centralized, governed context layer for agents, supporting file structures and connectors to integrate existing documentation (e.g., Notion, Slack, SharePoint). 2. Outcome Orientation: Specifying which metrics (deflection rate, CSAT, AHT) should be prioritized when agent improvements compete. 3. Dual Autopilot Tracks: Trigger-based runs for immediate error response and Hillclimbing Autopilot for regular audits focused on deflection and CSAT. 4. Outcome-based Metrics: Comparing Autopilot-merged changes against baseline metrics (CSAT, deflection) before and after shipping, with projected impact on deflection. 5. Duetbench-2: A formal benchmark to measure the durability of accepted changes over time (e.g., 85% of changes remain effective after 30 days).