A Byzantine failure in the real world

What this post added

This post details a significant incident involving a Byzantine fault in the etcd cluster due to a partial switch failure, which cascaded into database primary promotion issues and impacted API and dashboard availability. It highlights the challenges of handling degraded states in redundant systems, the limitations of RAFT in Byzantine scenarios, and the subsequent improvements made to database cluster management, user session handling, and auto-remediation configuration. It also touches upon the ongoing research into Byzantine Fault Tolerance (BFT) for production systems.