TechPrufer
Get started
Decagon
Decagon logoDecagon · Feature trail

AI Agent Testing and Simulation

2 posts · Feb 2026 – Jun 2026

What this trail covers

Decagon is enhancing its AI agent testing and simulation capabilities with Diagnostic Tools, which provides prescriptive insights to guide teams to the most important optimization opportunities. This includes Escalation Drivers to quantify when conversations escalate to a human agent and AOP Flow Analysis to understand conversation progression through Agent Operating Procedures. These features aim to accelerate agent improvements by transforming hillclimbing from manual guesswork into a data-dri. QA Hub provides a dedicated workspace for human-in-the-loop quality review, enabling teams to organize QA batches, apply custom rubrics, and track follow-ups. It integrates with Watchtower for automated QA and leverages Duet for sampling conversations, scoring interactions, and synthesizing reviewer feedback to identify improvement areas. The External QA Page allows for sharing standalone QA pages for testing.

All Decagon trails
  1. The next generation of Simulations: testing that keeps pace with your agents | DecagonThe latest generation of Decagon's Simulations introduces automated test generation from AOPs and production conversations using Duet, creating self-maintaining golden test sets that update automatically. It provides granular results through checkpoints and assertions, evaluating agent behavior across multiple dimensions. Simulations are now natively integrated into CI/CD pipelines for production-grade testing.
  2. Introducing Duet Autopilot: The self-improving agent for conversational AI | DecagonThis post introduces Duet Autopilot, an extension of the Duet platform that enables AI agents to self-improve. Autopilot automatically identifies issues from production signals, traces them to root causes, and proposes targeted fixes. It integrates with Duet's Simulations for automated testing and validation of proposed updates against original conversations and a curated golden test set. Proposed changes require human approval before deployment, with reviewers provided with detailed information on issues, changes, and validation results. Autopilot's improvements are fed back into its own operation, creating a compounding improvement loop.
  3. DuetBench: An evaluation of self-improving customer service agents | DecagonThis post introduces DuetBench, a novel evaluation framework for customer service AI agent self-improvement. It details the methodology for DuetBench, which includes simulated customer deployments, Agent Operating Procedures (AOPs), tools, guardrails, and synthetic traffic. The post presents results comparing Duet Autopilot against human baselines across diagnostic tasks (93% pass rate for Autopilot vs. 83% for humans), timed enterprise agent-building tasks (45.5% pass rate for Autopilot vs. 23% for humans within 3 hours, with Autopilot excelling on medium/hard tasks), test set validation (58% initial accuracy rising to 88% after self-critique), and end-to-end certification (Autopilot reaching the 90% human certification bar). The contribution is the creation and validation of this benchmark, demonstrating the capabilities of self-improving agents in a production-like environment.
  4. QA Hub: Agent quality is a team sport | DecagonThis post introduces QA Hub, a system for managing human-in-the-loop quality review for AI agents. It provides a dedicated workspace for organizing QA batches, applying custom rubrics, and tracking follow-ups. QA Hub integrates with Watchtower for automated QA and uses Duet to sample conversations, score interactions, and synthesize reviewer feedback to identify improvement areas. An External QA Page is also introduced for sharing standalone QA pages for testing.
  5. Introducing automatic optimization and Root Cause Analysis | DecagonThis post introduces Duet, an AI partner for Decagon's platform that provides automatic optimization and Root Cause Analysis for AI agents. Automatic optimization enhances Agent Operating Procedures (AOPs), brand guidelines, and guardrails by analyzing changes against conversation intents and surfacing opportunities to strengthen AOPs against best practices. Root Cause Analysis identifies patterns and generates targeted recommendations by analyzing low-scoring conversations from the Agent Performance Scorecard, clustering drivers of poor performance, and mapping them to specific components like AOP structure, knowledge gaps, and missing tools. Users can customize analysis by defining RCA goals and filtering by specifications.
  7. The evaluation engine behind Decagon’s AI agents | DecagonThis post details the evaluation engine behind Decagon's AI agents. It describes a two-phased framework: offline evaluation using LLM-as-judge (scoring relevance, correctness, naturalness, empathy) and ground truth evaluation, followed by online A/B testing with real customers. The system tracks impact on CSAT and resolution rate, monitoring for noise, novelty effects, and performance drift. Insights from all phases feed back into the experiment loop for prompt improvement, model upgrades, and agent behavior refinement. The evaluation benchmarks the entire agent architecture, including knowledge base retrieval, reasoning, and safety.
  8. Introducing Duet: Your agent building partner | DecagonThis post introduces Duet, an AI partner that automates agent building, testing, and optimization. Duet analyzes customer transcripts to generate AOPs, automatically creates test scenarios, simulates conversations to stress edge cases, and investigates live conversations to tune AOPs. It also analyzes data to generate custom reports highlighting trends and opportunities for improvement, enabling faster iteration and broader ownership of AI agent development.
  9. Agent Workbench: Debug AI agents autonomously | DecagonIntroduced Agent Workbench, an AI chat assistant integrated into the agent building process. Agent Workbench provides consolidated access to audit logs, agent reasoning, tool errors, and conversation history to accelerate debugging for business teams. It aims to enable autonomous identification of root causes, clear explanations of AI behavior, and actionable guidance for refining workflows and fixing issues, reducing reliance on engineering support.
  10. Climb higher: Introducing Diagnostic Tools for faster agent iteration | DecagonIntroduced Diagnostic Tools, featuring Escalation Drivers and AOP Flow Analysis, to accelerate AI agent iteration. Escalation Drivers quantify escalations by guardrail and AOP to prioritize refinements for better deflection rates. AOP Flow Analysis visualizes conversation progression through Agent Operating Procedures (AOPs) to identify unexpected drop-off points and refinement opportunities. Leverages Decagon's agent architecture with natural language instructions and structured nodes for precise analytics.
  11. Trace View: AI agents shouldn’t be black boxes | DecagonIntroduces Trace View, a new interface for observing the step-by-step execution of AI agents built on Agent Operating Procedures (AOPs). Trace View provides detailed visibility into LLM calls, tool invocations, condition evaluations, and safety checks, enabling granular debugging and performance analysis. It also includes latency tracing for each micro-step in the workflow, allowing for evidence-driven performance tuning. The post highlights how Trace View integrates with existing capabilities like Watchtower, AOPs, unit testing, and simulations to create a tight feedback loop for agent development and deployment.
  12. The future of AI agents is test-driven | DecagonThis post details the integration of real-time previews, unit tests, integration tests, and large-scale simulations into the Agent Operating Procedures (AOPs) development workflow. It highlights the importance of auditable agent execution paths for debugging and continuous iteration, aiming to bring predictability and transparency to AI agent development.
  13. Simulations: How AI agents get tested and trusted | DecagonThis post introduces 'Simulations,' a new feature within Decagon's AI agent testing and simulation capabilities. Simulations allow for the generation of test conversations using AI-driven mock users across various channels to thoroughly test an agent's core logic (AOPs), knowledge base retrieval, and guardrails. It emphasizes testing against actual failure modes derived from historical transcripts and simulating real-world voice conditions (accents, speaking speed, background noise, emotional tone). The feature integrates with Trace View for debugging and powers a continuous improvement loop by enabling automated QA and validation of fixes.