FOQS: Making a distributed priority queue disaster-ready
Jan 18, 2022Dillon George, Jasmit Kaur Saluja

Summary
This post details the migration of FOQS from a regional to a geo-distributed, global architecture to enhance disaster readiness. Key technical contributions include: 1. Architectural modifications to support routing and seamless failover of MySQL primaries across regions using global application primitives and Shard Manager. 2. Introduction of an intelligent routing service with an enqueue distributor for optimal data placement based on logical region preferences and a dequeue distributor for efficient item discovery. 3. Successful migration of tenants at scale with zero downtime for internal clients, addressing challenges like stuck queue items and high cross-region latencies inherent in the previous regional architecture.
More from the Site Reliability and Outage Management trail
- Lights Out, Systems On: Validating Instant Power Loss Readiness
- DrP: Meta’s Root Cause Analysis Platform at Scale
- Zoomer: Powering AI Performance at Meta’s Scale Through Intelligent Debugging and Optimization
- Diff Risk Score: AI-driven risk-aware software development
- How Meta keeps its AI hardware reliable