TechPrufer
Get started
Two Sigma
Two Sigma logoTwo Sigma · Feature trail

Data as Code

7 posts · Nov 2017 – Nov 2025

What this trail covers

Two Sigma has adopted a 'Data as Code' approach, applying software development principles like version control, automated testing, reproducibility, and CI/CD to data management. This involves using tools like Terraform for pipeline definition, dbt for SQL testing, and implementing automated data quality monitoring and lineage tracking. Migrating to BigQuery provided a scalable foundation, and internal tools were developed for anomaly detection, data discovery, and workflow orchestration. The integration of LLMs is further accelerating data preparation, transformation, and analysis, enabling natural language queries and automated documentation.

All Two Sigma trails
  1. Treating Data as Code at Two SigmaThis post details Two Sigma's adoption of the 'Data as Code' paradigm, emphasizing the application of software development principles (version control, automated testing, reproducibility, CI/CD) to data management. It highlights the use of Terraform for pipeline definition, dbt for SQL testing, and the development of internal tools for automated data quality monitoring, anomaly detection, data discovery, and workflow orchestration. The post also discusses the benefits of migrating to BigQuery for a scalable foundation and the emerging role of LLMs in accelerating data preparation and analysis.
  2. Improving Compute Sustainability: A Case StudyThis post details a case study where an engineering team improved compute sustainability by reconfiguring live trading application resources during a routine server refresh. The team consolidated underutilized legacy servers into denser, higher-core count hardware, reducing absolute power consumption by 66% and improving resilience and workload capacity. Key technical decisions involved selecting 28-core processor hardware, optimizing CPU and memory within established vendor capabilities, and enabling multiple trading programs to share a single cache of processed market data. This consolidation strategy aligns with green software patterns such as energy-efficient frameworks, optimizing average CPU utilization, and removing unused assets, demonstrating that sustainable engineering can be business-aligned.
  3. Semantic Types: From Computer-Centric to Human-Centric Data TypesThis post introduces the concept of Semantic Types for data, proposing a 'Semantic Enrichment of Data' (SED) initiative. It details use cases for semantic typing, including richer understanding of primitive types, improved data joining and linking (fuzzy joins), hierarchical and graph-oriented typing for knowledge graphs, and context-driven automatic validation. The post also outlines an implementation model for a Semantic Typing Data Model, emphasizing standardization of domain, formatting, conversion patterns, organization, and system characteristics, with a proposal for an open-source library.
  4. Diagnosing the Recent Decade of Drawdown in ValueThis post analyzes the historical performance of value investing strategies, specifically examining the book-to-price ratio's effectiveness in identifying value stocks. It highlights a regime change around the Global Financial Crisis (GFC) where the profitability and operating efficiency of stocks identified by the book-to-price metric diverged, with 'glamour' stocks becoming more profitable and 'value' stocks less so. This divergence is attributed to the increasing importance of intangible assets, which are not fully captured by traditional accounting book value. The analysis suggests that a value strategy incorporating intangibles, such as one based on earnings yield, has remained profitable and did not exhibit the same regime change, indicating that book value has become an incomplete fundamental valuation model for firms in the modern economy.
  5. Ibis and BigQuery: Scalable Analytics with the Comfort of PythonThis post introduces Ibis as a Python library that bridges the gap between the convenience of pandas and the scalability of SQL engines like BigQuery. It demonstrates how Ibis allows engineers to compose complex SQL queries using Python syntax, which are then compiled to SQL for execution on BigQuery. This approach aims to improve code readability, maintainability, and reduce common SQL errors like missing aliases.
  7. The Future of Database Management: An Interview with CMU’s Andy PavloThis post introduces the Succinct Range Filter (SuRF), a novel data structure that combines the fast point query capabilities of Bloom filters with approximate range query support. SuRF is engineered for performance and uses a succinct trie encoding to achieve low memory overhead (around 12 bits per 64-bit integer key for a 1% false positive rate). The post also discusses the potential long-term effects of ML on DBMS design and the trend towards shared-disk distributed DBMS architectures that can scan data in situ from various formats like Parquet, ORC, and JSON.
  8. Introducing Pandas UDFs for PySparkIntroduced Pandas UDFs (formerly Vectorized UDFs) for PySpark in Apache Spark 2.3. These UDFs leverage Apache Arrow to enable low-overhead, high-performance UDFs entirely in Python, addressing the serialization and invocation overhead of row-at-a-time UDFs. The post details two types: scalar Pandas UDFs (taking pandas.Series, returning pandas.Series) and grouped map Pandas UDFs (taking pandas.DataFrame, returning pandas.DataFrame). Examples include 'Plus One', 'Cumulative Probability', 'Subtract Mean', and 'Ordinary Least Squares Linear Regression'. Performance comparisons show 3x to over 100x improvement over row-at-a-time UDFs.