TechPrufer
Sign inGet started

Blogs›Datadog›Faulty Deployment Detection

Faulty Deployment Detection

Faulty Deployment Detection

1
posts
2025

Datadog developed a feature to automatically detect faulty deployments. The journey involved moving from unlabeled data to supervised learning using weak supervision. Initially, an iterative framework with statistical checks was used to identify deployments with increased error rates, considering impact, temporal correlation, and persistence. To improve time to detection, a sequence of models was introduced, running at different intervals (10, 20, and 60 minutes) after a deployment, each tuned for precision and recall at different stages of data accumulation. The system addresses challenges like lack of labels, data imbalance, and the diversity of application profiles.

2025

Detecting faulty deployments: Our journey from unlabeled data to supervised learning | Datadog

6/3/2025

This post details the evolution of Datadog's faulty deployment detection system. It describes the challenges of working with unlabeled data, data imbalance, and diverse application profiles. The initial approach involved an iterative framework with statistical checks and unanimous voting to identify faulty deployments based on error rate increases, impact, temporal correlation, and persistence. To address the time-to-detection trade-off, a sequence of models was implemented, running at 10, 20, and 60 minutes post-deployment, each tuned for precision and recall at different stages.