How Glean scales monitoring capability in the Cloud

This post details the development and features of Glean's internal Notification Dispatcher tool, designed to scale monitoring across multiple GCP projects. It introduces the tool's architecture, including an Error Reporting Crawler (App Engine Flex, Pub/Sub, Task Queue), Alert Notification Receiver (App Engine Flex, Pub/Sub), Notification Policy Rules (text proto), States Database (Cloud SQL), and the Notification Dispatcher execution engine. Advanced features like flexible Jira integration, robust incident/error deduplication using regex, custom notification silencing, frequency control, suppression of lower severity alerts, auto-escalation, and cross-project error grouping are highlighted. The tool addresses challenges of maintaining a global view of alerts and errors, integrating with SaaS applications like Jira, and enabling searchable knowledge from production issues.