The Consul outage that never happened

What this post added

This post details a near-outage on GitLab.com caused by expired TLS certificates for Consul, a service discovery tool. The incident highlighted the critical importance of certificate lifecycle management, especially for self-signed certificates used in proof-of-concept environments that are later promoted to production. The team successfully navigated the crisis by validating the problem, testing solutions in staging, and planning a 'break glass' approach using the `at` command for simultaneous service restarts across a large fleet of servers. The chosen solution involved temporarily disabling certificate validation to avoid downtime, with a plan to implement a more robust solution later.