Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale

What this post added

This post details the discovery and resolution of a metastable failure state in Facebook's network infrastructure related to aggregated links. The issue, which persisted for over two years, involved TCP traffic being imbalanced across aggregated links due to a hash-based routing strategy. The root cause was identified as an unintended consequence of a custom MySQL connection pool's Most Recently Used (MRU) policy, which, during query bursts, would preferentially select and keep congested links active. The fix involved switching to a Least Recently Used (LRU) connection pool with a maximum connection age, which stabilized network traffic and eliminated the metastable failure.