How Atlassian built a scalable synthetic data engine

This post introduces Data Brewery, Atlassian's synthetic data generator. It details the challenges of generating large, relational datasets for migration testing using existing API-based tools, highlighting API rate limits and slow entity coverage. The core technical contribution is the two-phase generation model: Phase 1 generates rows independently with unresolved foreign keys, and Phase 2 injects foreign keys in dependency order. This approach decouples row generation from relationship resolution, improving performance and scalability. The architecture is config-driven, with abstractions for product configuration, entity definitions, table definitions, field generators, and declarative relationships. The system scales to millions of rows by breaking jobs into smaller, independently processed chunks, enabling fault isolation and horizontal scaling.