TechPrufer
Get started
Two Sigma
Two Sigma logoTwo Sigma · Feature trail

Archival Storage System

1 post · Oct 2017

What this trail covers

Two Sigma developed CelFS, a geo-distributed file system for archival storage, which scaled to tens of petabytes but had high replication overhead and couldn't leverage outsourced cold storage. To address these limitations and reduce TCO, they developed Jaks, a new storage system designed to replace CelFS and serve as a backend for other systems. The development of Jaks involved hedging risks associated with changing such a foundational system.

All Two Sigma trails
  1. Archival Storage at Two SigmaThis post introduces Jaks, a new archival storage system developed to overcome the limitations of the existing CelFS system. CelFS, while scaled to petabytes, suffered from high replication overhead and an inability to utilize outsourced cold storage. Jaks aims to reduce the Total Cost of Ownership (TCO) of CelFS and serve as a backend for other systems. The post also discusses the strategies employed to hedge risk when modifying such a critical foundational system.