Building a Distributed Filesystem for Scalable Research

This post details the architecture of Blobby, HRT's custom distributed filesystem. It covers design principles (high aggregate throughput, durability, simplicity), data layout (replication and erasure coding with 6+3 default), client API (C++ library for metadata, read, write operations), data placement (Copysets and rack awareness), chunk server implementation (storing chunks as XFS files), metadata DB choice (FoundationDB for scalability, fault tolerance, ACID transactions, and strict serializability), proxy server roles (handling non-data operations and write steering), and background services (Shepherd for health checks and Repair for data recovery).