Minimizing on-call burnout through alerts observability

This post details the enhancement of alert observability by aggregating all alert states (firing, silenced, inhibited, and resolved) into a datastore using Vector.dev and ClickHouse. It describes the lifecycle of an alert within Prometheus and Alertmanager, the limitations of the previous alert reporting mechanism (alertmanager2es), and the proposed solution involving correlating data from Alertmanager webhooks and API endpoints using the fingerprint field. The implementation uses Vector.dev for data transformation and ClickHouse for storage, with a flexible schema to accommodate various alert labels. The outcome is multiple dashboards for comprehensive alert analysis.