Smooth Storage: A Distributed Storage System for Managing Structured Time Series Data at Two Sigma

This post discusses the design and implementation of Smooth, a distributed storage system for managing structured time series data at Two Sigma. It emphasizes scale, reliability, and storage efficiency, optimized for large parallel streaming read/write accesses over time ranges. Smooth features a separation between metadata and data layers, supports pluggable object stores, and allows data replication/movement for availability, performance, and tiering. It has been in development for 5 years, stores multiple PBs of compressed data, and serves peak aggregate throughput exceeding 100 GB/s. The author discusses experiences running it, ongoing challenges, and future directions.