Alki, or how we learned to stop worrying and love cold metadata
Dec 17, 2020Jonathan Lee

Summary
This post introduces Alki, a new petabyte-scale metadata store designed for cold metadata. It addresses the limitations of Edgestore, particularly its capacity expansion issues and unsuitability for high-volume, cold data like audit logs. Alki employs a two-tier architecture (hot and cold storage) based on LSM tree principles to optimize for different access patterns. The post details the motivations, architecture, and rapid prototyping/productionization of Alki.
More from the Magic Pocket Storage Efficiency trail
- Improving infrastructure efficiency for growing demand in the age of AI
- Improving storage efficiency in Magic Pocket, our immutable blob store
- Reducing our monorepo size to improve developer velocity
- Increasing Magic Pocket write throughput by removing our SSD cache disks
- Future-proofing our metadata stack with Panda, a scalable key-value store