Hudson River Trading Feature Trails

Feature trailActive

AI Labs Summer Projects

HRT AI Labs hosted interns who contributed to various AI research projects. Projects included Signal Architecture Search using Genetic Algorithms, optimizing Hyperparameter Search with Graybox Bayesian Optimization, and accelerating Quantitative Finance with GPU Computation by investigating CUDA performance for tall & skinny matrix multiplications. The interns' work focused on applying AI techniques to trading problems, exploring novel algorithms and performance optimizations. This post details the application of diffusion models to generate sequences of stock returns, exploring conditional and cross-sectional modeling, and defining evaluation metrics. It also discusses using market metadata as latent features for conditioning and inpainting capabilities. The post highlights diffusion models' potential for synthetic time-series data generation, avoiding mode collapse issues seen in autoregressive models.

2 posts · 2021–2026

Timeline

20212026

Feature trailActive

SmallGrid Real-time Data Publishing

SmallGrid is a tabular pub-sub system used for inter-process communication, GUI population, and infrastructure introspection. It supports efficient representation of market data and system metrics in a tabular format. This system has been enhanced to support compression for DenseColumnarTable, optimizing the trade-off between bandwidth usage and CPU overhead. Various compression algorithms (LZ4, LZ4Small, Deflate) are supported, with a focus on variable-size column packing to improve efficiency by grouping values by column and using minimal length tags. Null locations are encoded using explicit null bitmasks. The system also supports publishing to Redis Streams and Kafka topics, with APIs in C++ and Python.

2 posts · 2021–2026

Timeline

20212026

Feature trailActive

SystemVerilog Language Server

HRT developed and open-sourced slang-server, a SystemVerilog language server based on the Slang project. This provides intelligent code navigation, instant feedback, and features like go-to-definition and completion to editors like VS Code, Vim, and Emacs. It leverages Slang's parsing and elaboration passes to analyze SystemVerilog code, enabling faster and more accurate error checking and design understanding for hardware development workflows. The server's indexer has been optimized for latency and memory usage.

1 post · 2026

Timeline

Feature trailActive

SharedWorker Framework for Real-Time Trading GUIs

HRTWorker is a framework leveraging SharedWorker to enable efficient, shared data loading and processing across multiple browser tabs for internal real-time trading dashboards. This addresses challenges with duplicate data loading and complex communication patterns inherent in SharedWorkers by providing an abstract interface for developers to implement business logic. This post details the implementation of the framework, including connection management, an execution protocol for cross-context i

1 post · 2025

Timeline

20252026

Feature trailActive

HeraclesQL Alerting DSL

HeraclesQL is a Python DSL for writing VictoriaMetrics alerts, addressing code duplication, expressiveness, and footguns in MetricsQL. It generates type-safe MetricsQL, enabling reusable functions, parameterizable expressions, and static type checking to improve alert development and maintenance. This post details the technical implementation of the DSL, including its design principles, syntax, and how it generates MetricsQL. The DSL has been open-sourced on PyPI and GitHub.

1 post · 2025

Timeline

20252026

Feature trailCooling

Blobby Distributed Filesystem

HRT built and operates Blobby, a custom distributed filesystem designed to meet its specific needs for storing and analyzing vast amounts of data. Blobby addresses limitations in cost, scale, scalability, multi-tenancy, and control found in existing solutions. It employs a design based on prior art, focusing on high aggregate throughput, durability, and simplicity. Key architectural components include client libraries, chunk servers, a FoundationDB-backed metadata database, proxy servers, and background services like Shepherd and Repair. The system supports traditional replication and erasure coding for data durability, and utilizes FoundationDB for ACID-compliant, horizontally scalable metadata management. This post details the architecture, including data layout, client API, data placement strategies using Copysets, chunk server implementation, metadata DB choices, proxy server roles, and background services for health monitoring and recovery.

2 posts · 2021–2025

Timeline

20212026

Feature trailCooling

Python Lazy Imports

Hudson River Trading has fully adopted lazy imports across its Python monorepo, significantly improving script and notebook startup times and the efficiency of its distributed compute framework. This was achieved by forking CPython and implementing PEP 690's lazy import mechanism, which defers module resolution and evaluation until attributes are actually referenced. The migration involved addressing challenges with import side-effects, import ordering, and implicit transitive imports. HRT has a revised proposal for an explicit 'lazy' keyword in future PEPs.

2 posts · 2023–2025

Timeline

20232026

Feature trailCooling

Diffusion Models for Financial Time-Series Generation

This post continues the exploration of diffusion models for financial time-series generation, building upon previous work. It details the application of Large Language Models (LLMs) to financial news for sentiment analysis and prediction, and the use of Transformers for modeling both tick-by-tick order book events and cross-sectional return data. The LLM section discusses prompt engineering and the impact of model size and examples on accuracy. The Transformer section covers data representation. This post further breaks down the application of AI to trading into three subproblems: Prediction, Optimization, and Execution, detailing the challenges and approaches for each within the context of financial markets.

3 posts · 2021–2025

Timeline

20212026

Feature trailCooling

Manual Trading Platform Expansion

HRT has expanded its manual trading capabilities with ToucanServer, a gRPC and protobuf-based server implemented in C++. ToucanServer allows for more versatile human-driven trading by exposing order entry, state management, and position-tracking features. It utilizes a multithreaded architecture with thread-safe queues and client-specific mutexes to handle concurrent client requests and order processing. This enables the development of flexible front-end interfaces for manual trading operations. The Parrot system has been enhanced to use gRPC streaming for improved performance, reducing per-request overhead and enabling asynchronous order submission via Trio.

2 posts · 2022–2025

Timeline

20222026

Feature trailQuiet since 2024

Compiler Optimizations

This post continues the exploration of compiler optimizations, focusing on devirtualisation. It details how virtual function calls incur overhead through vtable lookups and indirect calls, impacting cache efficiency and branch prediction. The post explains how compilers can devirtualise calls when the object type is known at compile time, discussing compiler flags like -fwhole-program and -flto, and the use of the `final` keyword. It also includes a benchmark demonstrating the performance impact.

1 post · 2024

Timeline

20242026

Feature trailQuiet since 2024

Python Type Annotations and Static Analysis

Hudson River Trading has adopted Python type annotations and static analysis tools like mypy to improve code quality, maintainability, and developer productivity across its large and evolving Python codebase. This approach helps detect downstream issues early, reduces reliance on manual verification, and enhances the robustness of shared business logic.

1 post · 2024

Timeline

20242026

Feature trailQuiet since 2023

Third-Party Binary Patching

HRT developed a method for hot-patching third-party binary libraries to address urgent production bugs when vendor updates are not feasible. This involves understanding ELF binary formats, soname versioning, and utilizing tools like patchelf, with modifications to handle alignment issues. The initial approach of manual binary editing was found to be insufficient due to symbol versioning complexities. Patchelf was then used, but required modification to correctly handle ELF segment alignment.

1 post · 2023

Timeline

20232026

Feature trailQuiet since 2023

Exception Backtrace Enhancement

HRT has overhauled its internal backtrace library to provide a modern interface for fetching binary names, symbol names, source code locations, and snippets from captured frame pointers. This enhances the debugging experience for C++ exceptions, including propagation into Python via bindings and providing context from std::terminate handlers. The system supports configurable granularity levels to balance information detail with performance overhead, deferring time-consuming symbol resolution when.

1 post · 2023

Timeline

20232026

Feature trailQuiet since 2023

Python Codebase Dependency Analysis

HRT has developed a toolkit called 'Tangle Tools' to analyze the dependency graph of its large Python codebase. This system helps identify and refactor tangled dependencies, which cause performance issues with runtime imports and static analysis. The tools leverage Python's ast library, networkx for graph algorithms, and libcst for code transformations. This enables more efficient development iteration loops and reduces CPU waste in their distributed compute environment.

1 post · 2023

Timeline

20232026

Feature trailQuiet since 2023

Factor Models for Equity Returns

This post introduces factor models as a method for understanding and modeling equity returns. It details the mathematical formulation of linear factor models, explaining how asset returns can be decomposed into systematic factor exposures and idiosyncratic noise. The post discusses the benefits of factor models for covariance matrix estimation, reducing storage and computational complexity compared to empirical methods. It also explores their application in portfolio optimization by maximizing the Sharpe Ratio and decomposing performance attribution into parallel and orthogonal components.

1 post · 2023

Timeline

20232026

Feature trailQuiet since 2023

Trading UI Design System

HRT has developed a comprehensive design system for its internal trading UIs, focusing on user-centered design principles, tailored controls, visual consistency, and continuous feedback incorporation. This system aims to improve the efficiency and usability of tools used for monitoring trading performance, interacting with automated systems, and participating in manual trading operations. The Task Watcher tool is presented as a case study, demonstrating how user research, card sorting, and task analysis were used to redesign the UI for better usability and to establish a design system with consistent visual language and tailored controls. The system has been iterated on based on user feedback.

1 post · 2023

Timeline

20232026

Feature trailQuiet since 2022

Low Latency Optimization: Huge Pages

This post series explores low-latency optimization techniques, focusing on the role of huge pages and the Translation Lookaside Buffer (TLB) in the Linux operating system on x86 hardware. Part 1 explains the benefits of huge pages, detailing how they reduce memory management overhead and improve address translation latency by allowing the TLB to cover a larger memory footprint. A benchmark demonstrates significant performance improvements for memory-intensive operations when using huge pages. Part 2 will cover practical implementation and caveats in production environments.

2 posts · 2022

Timeline

20222026

Feature trailQuiet since 2022

Scalable Ethereum Reads

Introduced a method to significantly improve the speed and scalability of reading data from the Ethereum blockchain for research purposes. This involves modifying Ethereum client software (initially Erigon, then Geth) to operate in a read-only mode, allowing multiple clients to access a shared, frozen database. This eliminates the need for each client to run a full node or connect to a central server, enabling elastic scaling of read-only workloads and reducing operational overhead. The approach was extended to include tracing internal transactions using Geth's debug API, requiring a full sync and careful parsing of EVM state snapshots. The post details the challenges and solutions for extracting both top-level and internal transaction data into a queryable format.

1 post · 2021–2022

Timeline

20212026

Feature trailQuiet since 2022

Data Quality and Provenance Framework

HRT has developed a framework for evaluating and acquiring datasets for algorithmic trading. This framework focuses on heuristics such as relevance, uniqueness, avoiding lookahead bias, and assessing sample size and noise. It also outlines sources of data, including market data and alternative data, and discusses the creation of new datasets through data transformations. The importance of robust engineering practices for handling large datasets in production is also highlighted. This post elaborates on the heuristics for evaluating data quality, specifically focusing on relevance, uniqueness, avoiding lookahead bias, and assessing sample size and noise. It also details the sources of data, including market data and alternative data, and discusses the creation of new datasets through data transformations.

1 post · 2022

Timeline

20222026

Feature trailQuiet since 2021

FPGA Verification Environment

Hudson River Trading has developed a robust FPGA verification environment leveraging co-simulation. This environment integrates SystemVerilog simulation with Python-based testing frameworks like cocotb and Verilator for high-performance, in-process simulation. The approach emphasizes code reuse, developer productivity, and portability between simulation and lab environments, supported by continuous integration practices.

1 post · 2021

Timeline

20212026

Feature trailQuiet since 2021

Prometheus Scaling and Remote Read Architecture

HRT has developed a horizontally scalable Prometheus architecture by combining sharding with a remote read strategy. This involves dedicated query nodes that read from sharded ingestion nodes, enabling ad-hoc global aggregation and processing of massive recording rules. The system has been enhanced with high availability and redundancy using Consul for improved resilience. Concurrent remote read capabilities have been implemented upstream to improve query performance. This architecture allows HRT to ingest billions of samples per second while retaining global query capabilities.

1 post · 2021

Timeline

20212026