How groundcover built a durable alert dispatch system with Temporal

What this post added

This post details how groundcover built a durable alert dispatch system using Temporal's durable execution, SignalWithStartWorkflow, and per-alert Workflows to handle high-throughput fan-out and maintain ordering within alert streams. The system uses a two-level workflow architecture with an orchestrator and per-alert Workflows, signal batching for efficiency, and visibility API for backpressure management. State is managed in-memory within Workflows with persistence to PostgreSQL, leveraging Temporal's replay for durability. Retry policies are used for Activities, and notification delivery Activities handle external rate limits and retries.