Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale

This post details the discovery and resolution of a metastable link imbalance failure state in Facebook's aggregated network links. The root cause was identified as an interaction between a Most Recently Used (MRU) connection pool strategy in the TAO graph store and transient network congestion. The MRU pool, when faced with bursts of database queries, would preferentially select connections that had recently experienced latency (due to congestion), thereby exacerbating and sustaining the imbalance. The fix involved switching to a Least Recently Used (LRU) connection pool with a maximum connection age, which significantly improved traffic distribution and eliminated the metastable failure.