The Consul outage that never happened

This post details a near-outage incident caused by an expired self-signed TLS certificate for Consul. It describes the troubleshooting process, including validating the problem by temporarily disabling certificate validation and the subsequent cluster instability. The planning phase involved evaluating options like replacing the CA, using system store certificates, or disabling date validation. The team chose to disable date validation as an incremental step to avoid immediate downtime, demonstrating an iterative approach to security. The post also highlights the challenges of coordinating a simultaneous restart across a distributed team and the use of the `at` command for precise timing. The incident underscores the importance of production readiness reviews, monitoring, and robust certificate management practices.