TechPrufer
Get started
Hudson River Trading
Hudson River Trading logoHudson River Trading · Feature trail

Scalable Ethereum Reads

1 post · Dec 2021 – Mar 2022

What this trail covers

Introduced a method to significantly improve the speed and scalability of reading data from the Ethereum blockchain for research purposes. This involves modifying Ethereum client software (initially Erigon, then Geth) to operate in a read-only mode, allowing multiple clients to access a shared, frozen database. This eliminates the need for each client to run a full node or connect to a central server, enabling elastic scaling of read-only workloads and reducing operational overhead. The approach was extended to include tracing internal transactions using Geth's debug API, requiring a full sync and careful parsing of EVM state snapshots. The post details the challenges and solutions for extracting both top-level and internal transaction data into a queryable format.

All Hudson River Trading trails
  1. Scalable Ethereum ReadsThis post details the technical implementation of modifying Geth to operate in a read-only mode, enabling multiple instances to read from the same directory by disabling write attempts and file locking. It also describes the integration with Web3.py via CFFI and the creation of a custom provider to utilize a client-side RPC daemon compiled as a C shared library.
  2. Enabling Algo Research on Blockchain DataThis post details the extension of the scalable Ethereum reads capability to include the extraction and analysis of internal transactions. This involved setting up a Geth node with a full sync, utilizing the `debug_traceTransaction` API to capture EVM execution traces, and parsing these traces to reconstruct internal transaction data. A new table schema was defined for internal transactions, capturing details like call type, depth, input, and value. The post also discusses the challenges encountered with Geth's fast sync and the importance of thorough testing.