Two Sigma Feature Trails

Feature trailActive

Data as Code

Two Sigma has adopted a 'Data as Code' approach, applying software development principles like version control, automated testing, reproducibility, and CI/CD to data management. This involves using tools like Terraform for pipeline definition, dbt for SQL testing, and implementing automated data quality monitoring and lineage tracking. Migrating to BigQuery provided a scalable foundation, and internal tools were developed for anomaly detection, data discovery, and workflow orchestration. The integration of LLMs is further accelerating data preparation, transformation, and analysis, enabling natural language queries and automated documentation.

7 posts · 2017–2025

Timeline

20172026

Feature trailQuiet since 2024

Compute Carbon Intensity Measurement

Two Sigma is developing capabilities to measure and reduce the carbon intensity of its compute environment. This involves repurposing IPMI to measure server-level power consumption and integrating with Kubernetes (via Kubelet and Prometheus) to attribute power usage to containerized applications. The goal is to enable more efficient resource utilization, reduce operational costs, and mitigate environmental impact. This post details a case study where a trading engineering team leveraged a routine server refresh to reconfigure their resources in a way that meaningfully reduces power consumption while maintaining performance, consolidating deployments onto denser hardware configurations and reducing absolute power consumption by 66%.

1 post · 2023–2024

Timeline

20232026

Feature trailQuiet since 2022

Tensu: Text UI for Sensu Go Observability

Two Sigma developed Tensu, an open-source terminal-based dashboard for interacting with the Sensu Go observability pipeline. Tensu provides real-time data on monitored events, allowing administrators to quickly assess system status, filter events using regular expressions, and interact with checks (resolve, silence, re-execute) directly from the terminal. This reduces context switching and improves efficiency during production deployments and system changes. The post details the rationale and design approach for building Tensu, emphasizing its speed, ease of use, and interactive nature, and highlights its contribution to improving confidence in production deployments.

1 post · 2022

Timeline

20222026

Feature trailQuiet since 2022

Machine Learning for Thematic Equity Portfolios

Two Sigma is developing capabilities to construct thematic equity portfolios using machine learning. This involves analyzing historical equity sector and style factor behavior in different inflation environments, and then building supervised machine learning models (e.g., LightGBM) to predict stock returns based on various fundamental and factor loading data. The goal is to create portfolios that are designed to outperform in high inflation periods and underperform in low inflation periods, by focusing on features like relative employment size, industry, dividend payout ratio, and profit margin. The model is applied to construct a beta-neutral portfolio of individual stocks sensitive to inflation.

2 posts · 2021–2022

Timeline

20212026

Feature trailQuiet since 2021

Automated Migration Management System

Two Sigma developed ADMIRE (Adopt, Migrate, and Retire), a product-driven system for automating and managing system migrations. ADMIRE codifies migration scope, generates Jira tickets with specific instructions, tracks progress via dashboards, and automates ownership mapping. It has evolved to include dependency mapping for ticket sequencing and intelligent reminder systems. The system quantifies migration costs, informing further automation investments and driving organizational change towards automated migrations. This post details the evolution of ADMIRE to include automated dependency mapping and intelligent reminder systems, further reducing migration costs and improving efficiency.

1 post · 2021

Timeline

20212026

Feature trailQuiet since 2021

Machine Learning for Regime Modeling

Two Sigma is developing capabilities to model financial market regimes using machine learning. This involves applying Gaussian Mixture Models (GMMs) to historical factor return data to identify distinct market conditions (e.g., Crisis, Steady State, Inflation, Walking on Ice). The goal is to enable macroeconomically aware investment decision-making and better management of tail risks by understanding how portfolios react to different regimes. This post analyzes the recent decade of drawdown in value investing, attributing it to a regime change in the profitability and operating efficiency of stocks around the Global Financial Crisis (GFC). It highlights that traditional book-to-price based value strategies have underperformed because intangible assets, not captured by book value, have become significant drivers of profitability. The analysis suggests that a value strategy incorporating earnings yield, which accounts for intangibles, does not exhibit this regime change and has remained profitable.

1 post · 2020–2021

Timeline

20202026

Feature trailQuiet since 2021

Reinforcement Learning Scaling with RLlib and Ray

Two Sigma is developing capabilities to scale reinforcement learning (RL) training. This involves leveraging RLlib and its integration with Ray to distribute RL workloads across a cluster of commodity machines, significantly reducing training wall-time by overcoming CPU-bound bottlenecks in sample generation from environments. This approach enables orders of magnitude speedups in RL agent training. This post details how scaling RLlib with Ray can cut training wall-time down by orders of magnitude by removing CPU bottlenecks.

1 post · 2021

Timeline

20212026

Feature trailQuiet since 2021

Crypto Asset Risk Analysis

Two Sigma is developing capabilities to analyze the risk of crypto assets. This involves applying traditional financial risk factor models to understand Bitcoin's risk and its correlation with factors like Equity, Trend Following, and Emerging Markets. The analysis also explores common risk drivers among various cryptocurrencies using Principal Components Analysis and correlation matrices. The goal is to provide institutional investors with a framework for assessing crypto asset risks within a multi-asset class portfolio.

1 post · 2021

Timeline

20212026

Feature trailQuiet since 2020

Secure Boot and System Attestation

Two Sigma is developing capabilities to enhance the security of Linux boot processes on laptops and servers. This involves leveraging UEFI firmware features like Platform Key enrollment for kernel and initrd signing, and Trusted Platform Module (TPM) integration for protecting disk encryption keys. The system aims to prevent tampering with the boot process, detect persistence attempts, and provide remote attestation of system integrity. This post introduces 'safeboot', a package that makes it easy to take a

3 posts · 2017–2020

Timeline

20172026

Feature trailQuiet since 2019

Machine Learning Interpretability Methods

Two Sigma is developing capabilities to enhance the interpretability of machine learning models. This involves exploring and implementing various techniques to understand how models arrive at their predictions, moving beyond 'black box' behavior. The goal is to enable better debugging, bias detection, knowledge transfer, and assessment of trade-offs in ML systems. Methods discussed include Partial Dependence Plots (PDP), Individual Conditional Expectation (ICE), Permuted Feature Importance, Global Surrogates, and Local Surrogates (LIME). This post provides a brief survey of these methods, detailing their advantages and disadvantages.

1 post · 2019

Timeline

20192026

Feature trailQuiet since 2019

Automated Parameter Tuning Service Integration

Two Sigma is integrating external services like SigOpt for automated parameter tuning to accelerate research and development. This involves using advanced optimization techniques for complex simulations and machine learning algorithms where traditional methods are insufficient. The integration aims to improve performance, enable multi-metric optimization, and ensure efficient utilization of compute resources by leveraging asynchronous parallelization. The post details the integration of SigOpt, highlighting its ability to deliver results faster and cheaper than previous solutions, including multi-metric optimization and asynchronous parallelization.

1 post · 2019

Timeline

20192026

Feature trailQuiet since 2019

Automated Cloud Security Auditing

Two Sigma is enhancing its cloud security posture by automating security audits and compliance checks. This involves leveraging open-source tools like Forseti for Google Cloud Platform (GCP) to inventory cloud resources, define custom security rules, and detect violations. The system integrates with Google Cloud Security Command Center (Cloud SCC) to record security findings immutably and trigger notifications for remediation. This approach replaces manual, labor-intensive workflows with an agile and automated system. This post details the integration of Forseti with Cloud SCC via the Cloud SCC API to directly write violations, ensuring immutability and greater control over findings. A notification mechanism was also built to alert internal teams and suggest remediation steps.

1 post · 2019

Timeline

20192026

Feature trailQuiet since 2019

High-Throughput Metrics System

Two Sigma built a high-throughput metrics system to collect query-level usage data from a high-performance data platform. The system uses Elasticsearch for storage and analysis, Kafka as a buffer to handle bursts of queries, and Logstash to transfer data from Kafka to Elasticsearch. High availability is ensured through managed services for Kafka and Elasticsearch, and process management for Logstash (using Waiter). This post discusses engineering systems using open source solutions, highlighting the importance of careful selection and consideration of factors like support and community when choosing open source offerings. The system was presented at the Grace Hopper Celebration.

2 posts · 2018–2019

Timeline

20182026

Feature trailQuiet since 2018

Gradient Sparsification for Distributed Optimization

Two Sigma is developing capabilities in distributed optimization, focusing on reducing communication overhead. This post introduces gradient sparsification, a technique that randomly drops coordinates of stochastic gradients and amplifies the remaining ones to ensure unbiasedness, thereby minimizing communication costs in large-scale machine learning applications. Experiments on logistic regression, SVMs, and CNNs validate the approach.

1 post · 2018

Timeline

20182026

Feature trailQuiet since 2018

Gradient Sparsification for Distributed Optimization

1 post · 2018

Timeline

20182026

Feature trailQuiet since 2018

Flint: Time-Series Library for Apache Spark

Two Sigma developed Flint, an open-source library for Apache Spark that enhances time-series analysis capabilities. Flint focuses on time-series manipulation, offering functionalities like asof joins (with tolerance), adding columns for cycle-based computations (e.g., ranking, Box-Cox transformation), and various summarization functions (across entire dataframes, cycles, intervals, windows, and cumulative aggregations). It also provides windowing capabilities similar to Spark SQL's rolling windows. This post details the integration of Apache Arrow to improve Python and Spark performance and interoperability, enabling seamless and efficient use of big data tools without overhead by leveraging shared memory and avoiding serialization costs.

3 posts · 2017–2018

Timeline

20172026

Feature trailQuiet since 2018

Subgroup Mining Algorithm

Two Sigma developed MiSoSouP, a scalable algorithm for identifying interesting subgroups within datasets. It leverages statistical learning theory, specifically the concept of pseudodimension, to efficiently approximate subgroup distributions from a small random sample of the data. This approach significantly improves runtime and scalability compared to existing methods, enabling high-quality approximations with a fraction of the data. The algorithm was presented at ACM KDD'18 and invited for submission to the ACM Transactions on Knowledge Discovery from Data (TKDD) journal. This post details the algorithm's formulation, pseudodimension analysis, and empirical evaluation, demonstrating its superiority over state-of-the-art algorithms in terms of speed and quality of subgroup discovery.

2 posts · 2018

Timeline

20182026

Feature trailQuiet since 2018

Sundial: Distributed OLTP Database Concurrency Control and Caching

Two Sigma developed Sundial, an in-memory distributed optimistic concurrency control protocol for OLTP databases. Sundial addresses high latency and contention in distributed transactions by dynamically determining the logical order of transactions at runtime using logical leases on data elements, and by caching remote data locally to reduce access overhead. This unified protocol integrates concurrency control and cache coherence, outperforming existing protocols under high contention and significantly improving performance. This post details the research behind Sundial, including its design, implementation, and evaluation, and discusses its implications for future database systems.

1 post · 2018

Timeline

20182026

Feature trailQuiet since 2018

Machine Learning Bot Development for Halite

Two Sigma has developed capabilities to build and support machine learning bots for AI programming challenges like Halite. This involves providing ML starter kits, exploring different ML approaches (supervised learning vs. reinforcement learning), and establishing best practices for developing and evaluating ML bots in complex game environments. The focus is on practical application of ML, efficient data utilization, and robust performance evaluation.

1 post · 2018

Timeline

20182026

Feature trailQuiet since 2017

Waiter: Distributed Auto-Scaler

Two Sigma developed Waiter, an open-source distributed auto-scaler that provides request-level scaling for services. It supports service versioning and manages service lifecycles, automatically reaping inactive versions. Waiter has been used in production for over two years, handling hundreds of millions of requests per day. It is agnostic to underlying cluster managers and can run existing HTTP services without modification.

1 post · 2017

Timeline

20172026

Feature trailQuiet since 2017

Efficient Server Audit

Two Sigma is developing capabilities to verify the integrity of responses from untrusted concurrent servers, particularly in web application scenarios. This involves addressing the Efficient Server Audit Problem through techniques like simultaneous replay and efficient verification of concurrent executions. The goal is to provide a robust method for auditing server behavior and ensuring that responses are correctly derived from the executed program and client requests, with a focus on minimizing overhead for the server and achieving speedups over simple re-execution.

1 post · 2017

Timeline

20172026

Feature trailQuiet since 2017

Archival Storage System

Two Sigma developed CelFS, a geo-distributed file system for archival storage, which scaled to tens of petabytes but had high replication overhead and couldn't leverage outsourced cold storage. To address these limitations and reduce TCO, they developed Jaks, a new storage system designed to replace CelFS and serve as a backend for other systems. The development of Jaks involved hedging risks associated with changing such a foundational system.

1 post · 2017

Timeline

20172026

Feature trailQuiet since 2017

Highly Available OpenStack Networking

Two Sigma implemented a highly available networking architecture for its on-premise OpenStack private cloud. This involved using VXLAN as an overlay protocol for multi-rack connectivity, VRRP for redundant default gateways, and BGP for advertising routes to external networks. The system was designed to provide tenant networks with 'real' IPs, ensure no single points of failure, and achieve near bare-metal performance. Operational experience and tuning strategies for deployment, troubleshooting, performance, and IP space efficiency were also detailed.

1 post · 2017

Timeline

20172026

Feature trailQuiet since 2017

Head-to-Head Development System

Two Sigma is developing a 'Head-to-Head Development' system to manage software versioning and releases for a large number of developers and interconnected components. This system aims to address challenges in scalability, incrementality, and stability by focusing on efficient build and test processes, fine-grained dependency management, and robust infrastructure. The system also aims to foster cultural shifts towards shared code ownership and dependency trust. This post introduces the core principles of trunk-based development, monorepos, source integration, gated sharing, and implicit versioning as the foundation of their approach.

2 posts · 2017

Timeline

20172026

Feature trailQuiet since 2017

DPLL(T) String Solvers

Two Sigma is developing capabilities in formal verification and security applications by enhancing the scalability and efficiency of string solvers. This involves developing and implementing advanced techniques, such as context-dependent simplification, within SMT solvers like CVC4. The goal is to improve the ability to reason about complex string constraints arising from real-world problems, particularly in symbolic execution for security analysis.

1 post · 2017

Timeline

20172026

Feature trailQuiet since 2017

Dynamic Graph Triangle Counting

Two Sigma is developing capabilities for analyzing dynamic graphs, specifically focusing on efficiently counting triangles (local and global) in streams of edge insertions and deletions. This involves creating one-pass streaming algorithms that use reservoir sampling to maintain a fixed memory footprint while providing unbiased, low-variance approximations. The goal is to outperform existing methods in accuracy and update time on very large graphs. This post introduces TRIÈST, a suite of sampling algorithms for dynamic graph triangle counting, and Wiggins, an algorithm for detecting valuable information in dynamic networks with limited resources by optimizing a probing schedule through convex optimization and a scalable MapReduce variant.

3 posts · 2016–2017

Timeline

…20172026

Feature trailQuiet since 2017

GPU Non-Determinism Workaround

Two Sigma is developing capabilities to ensure repeatability in machine learning experiments, particularly when using GPUs. This involves identifying and mitigating sources of non-determinism in frameworks like TensorFlow, enabling more reliable debugging and experimentation.

1 post · 2017

Timeline

20172026

Feature trailQuiet since 2017

Graph Summarization Algorithms

Two Sigma is developing capabilities in graph summarization, focusing on creating concise, lossy representations of large graphs. These summaries enable faster approximate query answering for metrics like adjacency, degree, eigenvector centrality, and triangle/subgraph counting. The work involves developing polynomial-time approximation algorithms based on connections to geometric clustering (k-means, k-median) and exploring techniques for compressed graph representation. This post introduces ABRA, a suite of algorithms to compute and maintain probabilistically guaranteed, high-quality approximations of the betweenness centrality of all nodes (or edges) on both static and fully dynamic graphs, using progressive random sampling and analyses relying on Rademacher averages and pseudodimension. ABRA is much faster than exact methods and outperforms state-of-the-art algorithms with the same quality guarantees.

3 posts · 2016–2017

Timeline

…20172026

Feature trailQuiet since 2016

Fair Preemptive Resource Scheduler

Two Sigma developed Cook, a fair preemptive resource scheduler for compute clusters. Cook addresses the challenge of balancing latency-sensitive interactive jobs with throughput-oriented batch jobs by using a Cumulative Resource Share (CRS) metric to prioritize tasks and preempt lower-priority tasks when necessary. Benchmarks show that preemption significantly reduces latency for interactive jobs with a minimal impact on overall cluster utilization.

1 post · 2016

Timeline

…20172026

Feature trailQuiet since 2016

Deep Learning Research at ICML

Two Sigma is actively engaged in advancing deep learning research, with a focus on improving recurrent neural networks (RNNs) through techniques like unitary evolution and orthogonal weight matrices to address vanishing/exploding gradients. The company is also exploring unsupervised learning objectives to augment supervised training for large-scale image classification and deconstructing architectures like Ladder Networks. Furthermore, research is ongoing in supervised training methods, including dropout distillation and normalization propagation for better stability and performance, as well as gradient-based tuning of regularization hyperparameters. Deep reinforcement learning is also an area of focus, leveraging deep learning for complex tasks.

1 post · 2016

Timeline

…20172026

Feature trailQuiet since 2016

Open Community Runtime for Extreme Scale Computing

Two Sigma is developing capabilities for extreme-scale computing through the Open Community Runtime (OCR). OCR is a runtime system designed for power-constrained, highly distributed, and fault-prone environments. It minimally adds synchronization events and relocatable data blocks to support various programming models. The system aims to provide a flexible execution model for future computing paradigms.

1 post · 2016

Timeline

…20172026

Feature trailQuiet since 2016

Compiler Generation with HACS

Two Sigma is developing capabilities in compiler generation, focusing on creating a language (HACS) that simplifies the process of building fully functional compilers from a single source file. This system aims to make compiler construction more accessible and efficient, drawing on prior work in higher-order rewriting and abstract syntax.

1 post · 2016

Timeline

…20172026