Future-proofing our metadata stack with Panda, a scalable key-value store
Nov 3, 2022Preslav Le

Summary
Introduced Panda, a petabyte-scale transactional key-value store, as a new layer between Edgestore and sharded MySQL to address scalability and cost-effectiveness issues. Panda abstracts MySQL shards, supports ACID transactions, enables incremental capacity expansion, and allows for rebalancing of data. This architecture separates application and storage concerns, enabling independent iteration and potential unification of backends like Edgestore and Filesystem. Explored and rejected FoundationDB, Vitess, and CockroachDB due to limitations in scale, transaction support, consistency guarantees, and replication models.
More from the Magic Pocket Storage Efficiency trail
- Improving infrastructure efficiency for growing demand in the age of AI
- Improving storage efficiency in Magic Pocket, our immutable blob store
- Reducing our monorepo size to improve developer velocity
- Increasing Magic Pocket write throughput by removing our SSD cache disks
- Everything in its write place: Cloud storage abstraction with Object Store