Detecting faulty deployments: Our journey from unlabeled data to supervised learning | Datadog

What this post added

This post details the evolution of Datadog's faulty deployment detection system. It describes the challenges of working with unlabeled data, data imbalance, and diverse application profiles. The initial approach involved an iterative framework with statistical checks and unanimous voting to identify faulty deployments based on error rate increases, impact, temporal correlation, and persistence. To address the time-to-detection trade-off, a sequence of models was implemented, running at 10, 20, and 60 minutes post-deployment, each tuned for precision and recall at different stages.