Efficient, reliable cluster management at scale with Twine
Jun 6, 2019Kenny Yu, Chunqiang (CQ) Tang
Summary
This post details the evolution and architecture of Twine, Meta's cluster management system that orchestrates containers on millions of servers. It highlights key features such as seamless support for stateful services via the TaskControl interface, a single control plane across data centers using Resource Broker for dynamic server-scheduler binding and maintenance coordination, and transparent control plane sharding for scalability. The post also discusses the shift towards a global shared fleet and the challenges of managing competing requirements, emphasizing Twine's role in enabling elastic compute and efficient capacity shifting.
More from the Scalability and Large-Scale Systems Engineering trail
- Open-Sourcing Rebalancer: A Generic, High-Performance Library for Solving Assignment Problems
- ZGateway: Learnings from Putting a Proxy in Front of ZippyDB
- MTIA 300: Meta’s First Training Chip with Built-in NICs and Communication-Offloading Engines
- From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta’s Ads Ranking
- GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model