Scaling Prometheus to Billions of Samples a Second

This post details HRT's approach to scaling Prometheus beyond single-server limits by implementing a sharded ingestion model combined with a remote read architecture. It explains the rationale for deviating from community recommendations (federation) in favor of sharding for ad-hoc global aggregation. The post outlines the architecture with query nodes and sharded ingestion nodes, highlighting benefits like horizontal scalability for data collection, enabling massive recording rules, and reduced restart times. It also addresses challenges such as increased complexity and points of failure, detailing the implementation of high availability and redundancy using Consul. Finally, it discusses performance improvements achieved through upstream contributions for concurrent remote reads and notes limitations in remote read support for certain API endpoints.