Faulty Deployment Detection

Datadog developed a feature to automatically detect faulty deployments. The journey involved moving from unlabeled data to supervised learning using weak supervision. Initially, an iterative framework with statistical checks was used to identify deployments with increased error rates, considering impact, temporal correlation, and persistence. To improve time to detection, a sequence of models was introduced, running at different intervals (10, 20, and 60 minutes) after a deployment, each tuned differently. This post details the evolution from an unsupervised approach to a sequence of supervised models to improve time to detection and recall.