Treating Data as Code at Two Sigma

This post details Two Sigma's adoption of the 'Data as Code' paradigm, emphasizing the application of software development principles (version control, automated testing, reproducibility, CI/CD) to data management. It highlights the use of Terraform for pipeline definition, dbt for SQL testing, and the development of internal tools for automated data quality monitoring, anomaly detection, data discovery, and workflow orchestration. The post also discusses the benefits of migrating to BigQuery for a scalable foundation and the emerging role of LLMs in accelerating data preparation and analysis.