TechPrufer
Get started
Dropbox
Dropbox logoDropbox · Feature trail

Scalable Consistent Metadata Caching

1 post · Jul 2024

What this trail covers

Dropbox developed Chrono, a scalable and consistent caching system built on top of its key-value storage system, Panda. Chrono addresses the challenge of serving high-volume read QPS while upholding read-after-write consistency guarantees for metadata clients. It achieves this by decoupling write attempt tracking from data caching, allowing for independent scaling of components and ensuring that stale cache entries are detected and avoided. This enables new high-volume metadata workflows involving search and AI.

All Dropbox trails
  1. Meet Chrono, our scalable, consistent, metadata caching solutionThis post introduces Chrono, a new system designed to provide scalable and consistent metadata caching. It details the challenges of consistent caching at scale, particularly the difficulty in maintaining read-after-write guarantees when using traditional caching mechanisms. Chrono leverages Panda's MVCC capabilities, specifically its commit timestamps and snapshot read APIs, along with a new `Attempt` API to track write attempts. The system decouples write attempt tracking (Chrono Server) from data caching (Memcache), allowing for independent scaling and ensuring that stale cache entries are identified and avoided by using the latest attempt timestamp as an indicator. The write path involves notifying Chrono before writing to Panda, and the read path involves checking Memcache and potentially falling back to Panda if the cache entry is stale based on Chrono's timestamps.