TechPrufer
Get started
Datadog
Datadog logoDatadog · Feature trail

Autonomous SRE Agent Evaluation Platform

1 post · Sep 2024 – Apr 2026

What this trail covers

Datadog is evolving its autonomous SRE agent capabilities by building real-world evaluation platforms for agents like Bits Investigation. This platform enables reproducible investigation environments, using a curated label set of representative investigations and an orchestration platform to score agent performance. The goal is to address subtle regressions in agent behavior and scale the creation of evaluation labels by embedding it into the product. The platform now incorporates a pipeline for generating LLM-based postmortem drafts, integrating structured incident metadata with unstructured Slack discussions to produce accurate and reliable drafts while maintaining human author control. This involves an experimentation framework for data extraction and LLM instruction refinement, alongside LLM-generated incident summaries for rapid onboarding.

All Datadog trails
  1. How we built a real-world evaluation platform for autonomous SRE agents at scale | DatadogThis post details the development of a real-world evaluation platform for autonomous SRE agents, specifically Bits Investigation. It describes the need for reproducible investigation environments to detect subtle regressions, the limitations of tool-level testing and live replay, and the architecture of the platform which consists of a curated label set and an orchestration layer. Key contributions include the method for creating evaluation labels by embedding label creation into the product using user feedback and agent interactions, and the introduction of an agentic validation step to refine label quality before they are used for evaluation.
  2. How we optimized LLM use for cost, quality, and safety to facilitate writing postmortems | DatadogThis post details the development of an LLM-based feature within Bits AI to assist in writing postmortems. It describes the challenges of using LLMs in production, including data quality, hallucinations, cost/speed/quality trade-offs, and trust/privacy. The solution involves combining structured metadata from Datadog's Incident Management app with unstructured Slack discussions, feeding this into an ensemble of LLM models. Key technical efforts include establishing an experimentation framework for data extraction and LLM instruction refinement, using techniques like lowering model temperature and adjusting LLM parameters to reduce hallucinations and improve reliability. The post also mentions the development of LLM-generated incident summaries as a testing ground for rapid iterations on incident data.