TechPrufer
Get started
Dropbox
Dropbox logoDropbox · Feature trail

Magic Pocket Storage Efficiency

9 posts · May 2016 – Aug 2026

What this trail covers

Dropbox's Magic Pocket, an immutable blob store, faced increased storage overhead due to fragmentation caused by a new service that led to severely under-filled volumes. The existing L1 compaction strategy was insufficient to reclaim space efficiently from these sparse volumes. To address this, Dropbox developed a multi-strategy approach, including improvements to steady-state compaction and new methods for cleaning up the sparsest volumes, aiming to reduce overhead below previous baselines. This post details the new strategies for reclaiming space from under-filled volumes, including a new 'donor-less' compaction strategy and a targeted cleanup of the sparsest volumes, which successfully reduced storage overhead by 15% and improved effective replication.

All Dropbox trails
  1. Improving infrastructure efficiency for growing demand in the age of AIThis post details Dropbox's ongoing efforts to improve infrastructure efficiency, particularly in the context of growing AI demand. It expands on the Magic Pocket's storage efficiency by discussing broader strategies for capacity planning, fleet optimization (including Deep Sleep for power reduction and workload balancing), and increasing storage density. A key metric highlighted is the >50% improvement in watts per petabyte since 2020, demonstrating significant gains in power efficiency per unit of storage. The post also touches on hardware lifecycle management as a means to extend infrastructure utility.
  2. Improving storage efficiency in Magic Pocket, our immutable blob storeThis post details the development and implementation of new strategies to address storage fragmentation in Dropbox's Magic Pocket immutable blob store, specifically targeting severely under-filled volumes caused by a new service. The new strategies include: 1. Enhancements to steady-state compaction (L1) to improve efficiency. 2. A new 'donor-less' compaction strategy that consolidates live data from under-filled volumes into new, densely packed volumes without relying on existing host volumes. 3. A targeted cleanup process for the sparsest volumes, which involves migrating live data to new volumes and retiring the old ones. These changes successfully reduced storage overhead by 15% and improved effective replication.
  3. Reducing our monorepo size to improve developer velocityThis post details how Dropbox reduced its monorepo size from 87GB to 20GB by identifying and addressing an issue with Git's delta compression heuristic, which was inefficiently packing internationalization files due to their directory structure. By working with GitHub to implement a more aggressive repack strategy with tuned parameters, they significantly improved clone times and reduced operational risk.
  4. Increasing Magic Pocket write throughput by removing our SSD cache disksThis post details the removal of SSD cache disks from Dropbox's Magic Pocket storage system. Previously, live writes were committed to SSDs as a write-back cache before being asynchronously flushed to SMR disks. This architecture created a bottleneck, limiting host write throughput to the maximum write throughput of the single SSD. The team re-architected the system to write directly to SMR disks, storing block metadata and raw block data inline within a new extent format. This change increased write throughput by 15-20%, improved reliability by removing a single point of failure (SSD failures), and reduced operational overhead and costs associated with sourcing and maintaining SSDs. While p95 latencies for disk writes increased slightly, overall end-to-end write latency saw a minimal increase, and read latency for recently written blocks was impacted but not significantly at the p99 level.
  5. Future-proofing our metadata stack with Panda, a scalable key-value storeIntroduced Panda, a petabyte-scale transactional key-value store, as a new layer between Edgestore and sharded MySQL to address scalability and cost-effectiveness issues. Panda abstracts MySQL shards, supports ACID transactions, enables incremental capacity expansion, and allows for rebalancing of data. This architecture separates application and storage concerns, enabling independent iteration and potential unification of backends like Edgestore and Filesystem. Explored and rejected FoundationDB, Vitess, and CockroachDB due to limitations in scale, transaction support, consistency guarantees, and replication models.
  7. Everything in its write place: Cloud storage abstraction with Object StoreThis post details the design and implementation of Object Store, an abstraction layer over multiple cloud storage backends (S3, HDFS, and internally Magic Pocket). It addresses cost inefficiencies of using S3 by batching small PUT requests into larger blobs, reducing API request costs. It also introduces per-object encryption using AES-256 with a unique block encryption key (BEK) for each object, which is itself encrypted by a global key encryption key (KEK) for secure key rotation. For deletions, it implements crypto-shredding, where encrypting the BEK allows for effective deletion by simply removing the encrypted BEK from metadata, without requiring immediate data purging from the underlying store.
  8. Alki, or how we learned to stop worrying and love cold metadataThis post introduces Alki, a new petabyte-scale metadata store designed for cold metadata. It addresses the limitations of Edgestore, particularly its capacity expansion issues and unsuitability for high-volume, cold data like audit logs. Alki employs a two-tier architecture (hot and cold storage) based on LSM tree principles to optimize for different access patterns. The post details the motivations, architecture, and rapid prototyping/productionization of Alki.
  9. SMR: What we learned in our first yearThis post details Dropbox's first year of adopting Shingled Magnetic Recording (SMR) technology for its Magic Pocket storage. Key contributions include: implementing software changes such as an SSD staging layer to handle SMR's sequential write limitations, leveraging the immutable block architecture for compatibility, and collaborating with hardware vendors to adapt chassis and firmware. The team also contributed to open-source libraries (libzbc) and developed a testing tool (SMRtest). The adoption resulted in increased storage density (surpassing 25% SMR capacity goal), significant cost savings (10-20% more data per drive, >20% overall savings), and energy efficiency improvements.
  10. Inside the Magic PocketThis post provides a high-level architectural overview of Magic Pocket, Dropbox's immutable block storage system for file content. It details the system's design requirements: immutable block storage (up to 4MB chunks), workload characteristics (high temporal locality, low-latency reads), durability (effectively infinite theoretical durability via erasure coding and multi-region replication), scale (transition from double-digit petabytes to multi-exabytes), and simplicity (avoiding complex consensus). The data model is explained: blocks (identified by SHA-256 hash) are aggregated into 1GB buckets, which are then stored in volumes across replicated or erasure-coded physical nodes. The multi-zone architecture is described, with frontends, a sharded MySQL Block Index service, cross-zone replication daemons, and cells (logical storage clusters of ~50PB).