TechPrufer
Get started
Hudson River Trading
Hudson River Trading logoHudson River Trading · Feature trail

SmallGrid Real-time Data Publishing

2 posts · Sep 2021 – Aug 2026

What this trail covers

SmallGrid is a tabular pub-sub system used for inter-process communication, GUI population, and infrastructure introspection. It supports efficient representation of market data and system metrics in a tabular format. This system has been enhanced to support compression for DenseColumnarTable, optimizing the trade-off between bandwidth usage and CPU overhead. Various compression algorithms (LZ4, LZ4Small, Deflate) are supported, with a focus on variable-size column packing to improve efficiency by grouping values by column and using minimal length tags. Null locations are encoded using explicit null bitmasks. The system also supports publishing to Redis Streams and Kafka topics, with APIs in C++ and Python.

All Hudson River Trading trails
  1. Intern Spotlight: 2025 Software Engineering Summer ProjectsThis post details the addition of compression support to SmallGrid's DenseColumnarTable. It introduces the trade-offs between compression ratio and CPU overhead for algorithms like LZ4 and Deflate. A key technical contribution is the implementation of variable-size column packing, which groups values by column and uses minimal length tags to reduce per-cell metadata overhead and improve compression ratios. This packing also reduces total message count by aggregating updates and simplifies consumer-side decoding by using explicit null bitmasks. Benchmarks demonstrate significant improvements in both data size and CPU time.
  2. An Introduction to SmallGridThis post details the API for writing data to SmallGrid, including optimizations for row/column lookups using handles and heterogeneous lookup in hash maps. It also describes the implementation of a monitoring system that can automatically watch variables within an event loop, with support for C++ and Python APIs. The Python API is enhanced with a `toDataFrame()` method for easier testing and bulk publishing functions like `publishRow()` and `publishCol()`.