TechPrufer
Get started
Dropbox
Dropbox logoDropbox · Feature trail

Sync Engine Rewrite and Testing

4 posts · Mar 2020 – Sep 2020

What this trail covers

Dropbox's core sync engine, codenamed Nucleus, has undergone a significant rewrite to improve testability and robustness. This involved redesigning the protocol and data model to eliminate invalid states and simplify state representation. The new concurrency model, primarily single-threaded with offloaded I/O, enables deterministic randomized testing. This post details the architectural changes made to the sync engine to facilitate comprehensive automated testing, including randomized testing for performance regressions. The Apogee framework was developed to measure performance metrics (latency, CPU usage, disk I/O, memory usage, and network I/O) during end-to-end integration tests, aggregating and graphing results to detect performance regressions. Efforts to reduce measurement variability included static sharding of tests, running in Dropbox headless mode, using stage server variants, homogenous VMs with dedicated resources, and RAM disks for storage. This capability evolves to ensure the performance of the sync engine doesn't suffer during rollouts and to validate performance improvements.

All Dropbox trails
  1. The software engineering lifecycle: How we built the new Dropbox PlusThis post details the author's experience contributing to the development of the new Dropbox Plus features, specifically focusing on the Family plan and the Vault secure folder. For the Family plan, the author led frontend architecture, emphasizing independent application development for faster iteration and the use of API-QL for improved developer experience with REST endpoints. The post also touches on the transition to working from home due to COVID-19 and onboarding onto the Vault team. While the post discusses the software engineering lifecycle and team organization, it does not introduce new technical details about the sync engine itself, but rather contextualizes the development process around existing capabilities like the sync engine rewrite.
  2. Keeping sync fast with automated performance regression detectionThis post introduces Apogee, a performance regression testing framework for Dropbox's sync engine. Apogee measures latency, CPU usage, disk I/O, memory usage, and network I/O during end-to-end integration tests. It addresses measurement variability through techniques like static sharding of tests, running in headless mode, using stage server variants, homogenous VMs with dedicated resources, and RAM disks for storage. The framework's data pipeline involves CI, an aggregation service, InfluxDB, and Grafana.
  3. Testing sync at DropboxThis post details the architectural changes made to the Nucleus sync engine to improve testability. Key changes include designing away invalid system states by enforcing strict invariants in the protocol and data model (e.g., no orphaned files, single atomic moves for folder renames), and adopting a single-threaded concurrency model with offloaded I/O to enable deterministic randomized testing. The post also introduces the concept of randomized testing with a focus on determinism and reproducibility, using a pseudorandom number generator (PRNG) with a seed to ensure consistent test runs.
  4. Rewriting the heart of our sync engineThis post details the rationale and challenges behind rewriting Dropbox's core sync engine (Nucleus). It highlights issues with the legacy engine's data model, lack of testability, and difficulties in scaling. Key improvements in Nucleus include a cleaner data model, a single-threaded concurrency model with offloaded I/O for deterministic testing, and strategies to address durability and specification challenges in distributed file synchronization.