Cassandra Distributed Storage System

4 posts · Aug 2008 – Aug 2021

What this trail covers

Meta's infrastructure for storing and serving user content has evolved significantly, starting with photos and expanding to encompass immersive media like 360 video and VR. This evolution includes developing dedicated mobile applications, enhancing user experience for capturing and sharing, and tackling complex technical challenges in processing and delivering new media formats. Key developments include the creation of custom video filters for transforming equirectangular layouts to cube maps, a

All Meta trails

Aug 2021How we built a general purpose key value store for Facebook with ZippyDBThis post details the history, architecture, data model, and consistency guarantees of ZippyDB, Meta's largest strongly consistent, geographically distributed key-value store. It highlights the reuse of existing infrastructure like RocksDB, Data Shuttle, Shard Manager, and ZooKeeper, and introduces concepts like tiers, shards, μshards, and Akkio mapping. The post also explains ZippyDB's replication strategies (Paxos, async), consistency levels (eventual, read-your-writes, strong), and support for transactions and conditional writes.
Jun 2018Migrating Messenger storage to optimize performanceMigrated Messenger's storage from HBase to MyRocks (integrating RocksDB as a MySQL storage engine) and from spinning disks to flash storage on new Lightning Server SKUs. This involved redesigning the data schema, creating a new source-of-truth index, and implementing consistent invariants. Two migration flows (normal and buffered) were developed to handle over a billion accounts seamlessly, ensuring data consistency and API equivalence throughout the process. The migration resulted in a 90% reduction in storage consumption, significantly lower read latency (50x), improved system resiliency, and enabled easier addition of new features.
Apr 2015Parse now running MongoDB on RocksDBThis post details the integration of RocksDB as a storage engine for MongoDB within the Parse platform. It highlights significant improvements in write performance (e.g., 50x faster inserts) and compression (70%-90%), enabling more indexes to fit into RAM and thus improving overall system speed. This adoption enhances MongoDB's existing strengths in automatic failovers and workload flexibility.
Nov 2013Under the Hood: Building and open-sourcing RocksDBThis post introduces and open-sources RocksDB, an embeddable, persistent key-value store designed for high performance on modern hardware. It addresses challenges with traditional remote procedure calls by enabling direct access to fast storage, reducing network latency. RocksDB is built on LevelDB, focusing on scalability for multi-core servers, efficient storage utilization (IOPS, compression, write wear), and flexibility. It employs a log-structured merge-tree architecture with pluggable components for compression, compaction, and storage formats. Performance benchmarks show significant improvements over LevelDB, particularly for random writes and bulk uploads. Typical workloads include user-facing applications, spam detection, graph search, and message queues. The post details its architecture, performance characteristics, and suitable use cases, making the code available on GitHub.
Nov 2010The Underlying Technology of MessagesThis post details the migration of Meta's Messages infrastructure from a combination of MySQL and Cassandra to Apache HBase. The decision was driven by HBase's superior scalability and performance for handling both volatile temporal data and ever-growing, rarely accessed data, as well as its simpler consistency model compared to Cassandra. The post highlights the evaluation process, the choice of HBase over MySQL and Cassandra, and the integration with HDFS for underlying filesystem features. It also mentions the development of a custom application server and interfaces with other services like Haystack and ZooKeeper.
Advertisement
Aug 2008Cassandra – A structured storage system on a P2P NetworkThis post introduces Cassandra, a distributed storage system designed for massive scale and high availability. It details Cassandra's data model (key-value, column families, supercolumns, columns with timestamps) and its distributed properties, including consistent hashing for data distribution and range scans, gossip-style membership, and accrual-style failure detection for cluster management. It highlights replication for high availability, eventual consistency, and incremental scalability through node addition/removal. The post also mentions its initial deployment for Inbox Search at Facebook, storing TBs of data across hundreds of cores and terabytes of disk space, and its open-sourcing.