GitLab

How we improved on-call life by reducing pager noise

Jul 19, 2022Steve Azzopardi
How we improved on-call life by reducing pager noise
SummaryAI, from the source post

Introduced alert grouping in Alertmanager using the 'type' and 'env' labels to consolidate multiple SLI alerts for a single service into one notification. Implemented Alertmanager inhibition rules based on service dependencies defined in the metrics-catalog to prevent cascading alerts from downstream services. This reduced the number of pages for on-call engineers during service degradations and site-wide outages.

More from the Log Management and Observability trail