TechPrufer
Get started
Dropbox
Dropbox logoDropbox · Feature trail

Async Task Framework

2 posts · Nov 2020 – Jan 2025

What this trail covers

Dropbox's asynchronous platform has evolved significantly, moving from a collection of disparate, difficult-to-manage systems to a unified, event-driven architecture. This evolution, guided by a Messaging System Model (MSM), addresses challenges in developer efficiency, reliability, operability, scalability, and extensibility. The platform now handles over 30 million tasks per minute, supports AI innovations like Dropbox Dash, and integrates Change Data Capture (CDC) use cases. The MSM standardizes the platform into five logical layers, simplifying interfaces and enabling consistent delivery guarantees and data sources.

All Dropbox trails
  1. Evolving our infrastructure through the messaging system model in DropboxThis post details the evolution of Dropbox's asynchronous platform, addressing past challenges in scalability, reliability, developer efficiency, operability, and extensibility. It introduces the Messaging System Model (MSM), inspired by the OSI model, which divides the platform into five logical layers to standardize interfaces and improve consistency. The MSM enables the platform to handle over 30 million tasks per minute, support AI innovations like Dropbox Dash, and integrate Change Data Capture (CDC) use cases. The post outlines the key limitations of the previous infrastructure and the goals for the new platform, emphasizing development velocity, a robust and extensible async foundation, and cost/operational efficiency.
  2. How we designed Dropbox’s ATF - an async task frameworkThis post introduces Dropbox's Async Task Framework (ATF), detailing its design, architecture, and guarantees. It addresses the need for a unified async system to handle diverse tasks at scale. Key technical contributions include the callback-based architecture, use of gRPC and Edgestore, SQS for queuing, and the roles of Frontend, Store Consumer, Controller, and Executor. Guarantees like at-least-once execution and no concurrent execution are highlighted, along with lambda requirements for idempotence and resiliency. The post aims to fill an information gap regarding supersized async service development.