Detecting faulty deployments: Our journey from unlabeled data to supervised learning | Datadog

This post details the evolution of Datadog's faulty deployment detection system. It describes the challenges of unlabeled data and data imbalance, leading to an iterative unsupervised framework with statistical checks. The system was then enhanced by introducing a sequence of models running at different intervals (10, 20, and 60 minutes) after a deployment, each tuned differently to manage the trade-off between time to detection, precision, and recall. This iterative approach improved the system's ability to identify faulty deployments more promptly and accurately.