How we built a real-world evaluation platform for autonomous SRE agents at scale | Datadog

What this post added

This post details the creation of a real-world evaluation platform for Datadog's autonomous SRE agents. The platform addresses the problem of subtle regressions by enabling reproducible investigation environments. Key components include a curated label set (ground truth + world snapshot of signals) and an orchestration layer for execution and scoring. The post highlights the evolution of label creation from manual efforts to an embedded product pipeline, leveraging customer feedback and investigation data. It also touches upon the need for agentic validation of labels.