How we improved on-call life by reducing pager noise
Jul 19, 2022Steve Azzopardi

Summary
Introduced alert grouping in Alertmanager using the 'type' and 'env' labels to consolidate multiple SLI alerts for a single service into one notification. Implemented Alertmanager inhibition rules based on service dependencies defined in the metrics-catalog to prevent cascading alerts from downstream services. This reduced the number of pages for on-call engineers during service degradations and site-wide outages.
More from the Log Management and Observability trail
- Green DevOps: Why carbon measurement belongs in your CI/CD pipeline
- Automate detection testing with GitLab CI/CD and Duo
- How to build CI/CD observability at scale
- Track vulnerability remediation with the updated GitLab Security Dashboard
- More granular product usage insights for GitLab Self-Managed and Dedicated