Maintaining large-scale AI capacity at Meta

This post details Meta's strategies for maintaining large-scale AI capacity, specifically focusing on GPU training clusters. It highlights the challenges of ensuring capacity guarantees, handling 'bad hosts,' minimizing interruption rates, ensuring rollout safety, and maintaining host consistency in a dynamic environment. The post introduces 'maintenance trains' for cyclic server maintenance and 'gradual rollouts' for software and firmware updates, emphasizing the need for careful testing and vendor collaboration. It also describes the role of OpsPlanner in orchestrating disruptive work and ensuring host consistency, along with safety features like autostop and automatic offboarding of failing upgrades.