Containerizing ZooKeeper with Twine: Powering container orchestration from within

This post details the migration of all ZooKeeper deployments to Meta's internal container orchestrator, Twine. It describes the challenges of running a stateful, quorum-based system like ZooKeeper on a platform designed for stateless services, particularly the Twine Shared Pool. Key technical contributions include the development of a 'task controller' to enforce ZooKeeper's durability and quorum requirements on Twine's scheduler, and the engineering of 'break-glass' recovery techniques to mitigate circular dependencies with other Twine-powered infrastructure services (e.g., service discovery, configerator, fbpkg). The post highlights the benefits of this migration, including reduced operational toil, improved reliability through leveraging standard infrastructure features, and the elimination of years of obsolete custom code.