TechPrufer
Get started
Hudson River Trading
Hudson River Trading logoHudson River Trading · Feature trail

Prometheus Scaling and Remote Read Architecture

1 post · Oct 2021

What this trail covers

HRT has developed a horizontally scalable Prometheus architecture by combining sharding with a remote read strategy. This involves dedicated query nodes that read from sharded ingestion nodes, enabling ad-hoc global aggregation and processing of massive recording rules. The system has been enhanced with high availability and redundancy using Consul for improved resilience. Concurrent remote read capabilities have been implemented upstream to improve query performance. This architecture allows HRT to ingest billions of samples per second while retaining global query capabilities.

All Hudson River Trading trails
  1. Scaling Prometheus to Billions of Samples a SecondThis post details HRT's approach to scaling Prometheus beyond single-server limits by implementing a sharded ingestion model combined with a remote read architecture. It explains the rationale for deviating from community recommendations (federation) in favor of sharding for ad-hoc global aggregation. The post outlines the architecture with query nodes and sharded ingestion nodes, highlighting benefits like horizontal scalability for data collection, enabling massive recording rules, and reduced restart times. It also addresses challenges such as increased complexity and points of failure, detailing the implementation of high availability and redundancy using Consul. Finally, it discusses performance improvements achieved through upstream contributions for concurrent remote reads and notes limitations in remote read support for certain API endpoints.