Building a Multi Region Compliant Customer Data Lake at Scale

This post details the architecture and implementation of Atlassian's customer data lake, focusing on the challenges of data replication and streaming ETL. It describes the decision to use logical replication over physical replication for data capture, the design of a 'replication bus' using Protobuf for event standardization and S3 offloading for large payloads, and the use of Apache Flink and Delta Lake for building a streaming ETL pipeline. The system aims to provide instant analytics, a data export path for cloud customers, and offload analytical workloads from operational databases.