TechPrufer
Get started
Two Sigma
Two Sigma logoTwo Sigma · Feature trail

High-Throughput Metrics System

2 posts · Sep 2018 – Jan 2019

What this trail covers

Two Sigma built a high-throughput metrics system to collect query-level usage data from a high-performance data platform. The system uses Elasticsearch for storage and analysis, Kafka as a buffer to handle bursts of queries, and Logstash to transfer data from Kafka to Elasticsearch. High availability is ensured through managed services for Kafka and Elasticsearch, and process management for Logstash (using Waiter). This post discusses engineering systems using open source solutions, highlighting the importance of careful selection and consideration of factors like support and community when choosing open source offerings. The system was presented at the Grace Hopper Celebration.

All Two Sigma trails
  1. Building a High-Throughput Metrics System Using Open Source SoftwareThis post details the design and implementation of a high-throughput metrics system using open-source components: Elasticsearch for storage and analysis, Kafka as a buffer for high throughput, and Logstash for data transfer. It outlines the problem-solution approach, including criteria for selecting open-source tools, and addresses challenges in storage, throughput handling, data transfer, and high availability. The system was built to collect query-level usage data from a high-performance data platform without impacting its performance.
  2. Engineering with Open SourceThis post details the engineering considerations for building a high-throughput metrics system for a high-performance data platform using open-source solutions. It highlights the use of Elasticsearch for storage and analysis, Kafka as a buffer for query bursts, and Logstash for data transfer. The importance of selecting open-source tools carefully, considering factors beyond just functionality, is emphasized. The post was presented at the Grace Hopper Celebration.