How we built a real-world evaluation platform for autonomous SRE agents at scale | Datadog

This post details the development of a real-world evaluation platform for autonomous SRE agents, specifically Bits Investigation. It describes the need for reproducible investigation environments to detect subtle regressions, the limitations of tool-level testing and live replay, and the architecture of the platform which consists of a curated label set and an orchestration layer. Key contributions include the method for creating evaluation labels by embedding label creation into the product using user feedback and agent interactions, and the introduction of an agentic validation step to refine label quality before they are used for evaluation.