TechPrufer
Get started
Two Sigma
Two Sigma logoTwo Sigma · Feature trail

Flint: Time-Series Library for Apache Spark

3 posts · Jun 2017 – Oct 2018

What this trail covers

Two Sigma developed Flint, an open-source library for Apache Spark that enhances time-series analysis capabilities. Flint focuses on time-series manipulation, offering functionalities like asof joins (with tolerance), adding columns for cycle-based computations (e.g., ranking, Box-Cox transformation), and various summarization functions (across entire dataframes, cycles, intervals, windows, and cumulative aggregations). It also provides windowing capabilities similar to Spark SQL's rolling windows. This post details the integration of Apache Arrow to improve Python and Spark performance and interoperability, enabling seamless and efficient use of big data tools without overhead by leveraging shared memory and avoiding serialization costs.

All Two Sigma trails
  1. Introducing Flint: A Time-Series Library for Apache SparkThis post introduces Flint, an open-source library for Apache Spark designed to enhance time-series manipulation capabilities. It details core functionalities such as Asof Join (with tolerance for inexact time matching), AddColumnsForCycle (for transformations within the same timestamp), Summarizers (for aggregations across dataframes, cycles, intervals, windows, and cumulative sums), and Window functions (for rolling computations). A case study demonstrates data preparation, feature engineering using Flint, and model training with Spark ML, highlighting the library's utility in handling time-series data with varying frequencies and misaligned timestamps.
  2. Smooth Storage: A Distributed Storage System for Managing Structured Time Series Data at Two SigmaThis post discusses the design and implementation of Smooth, a distributed storage system for managing structured time series data at Two Sigma. It emphasizes scale, reliability, and storage efficiency, optimized for large parallel streaming read/write accesses over time ranges. Smooth features a separation between metadata and data layers, supports pluggable object stores, and allows data replication/movement for availability, performance, and tiering. It has been in development for 5 years, stores multiple PBs of compressed data, and serves peak aggregate throughput exceeding 100 GB/s. The author discusses experiences running it, ongoing challenges, and future directions.
  3. Improving Python and Spark Performance and Interoperability with Apache ArrowThis post details the integration of Apache Arrow with Python and Spark to improve performance and interoperability. It explains how Arrow enables efficient data exchange between Python and Spark by avoiding serialization costs through shared memory and scatter-gather I/O, addressing performance bottlenecks in PySpark jobs and costly data transfers between Python environments and Spark masters.