TechPrufer
Get started
Meta
Meta logoMeta · Feature trail

Data Warehousing and Analytics Platform

109 posts · Jun 2008 – Jul 2026

What this trail covers

Meta's data management systems have evolved into a composable architecture, leveraging reusable libraries and common APIs to improve interoperability, promote reusability, and enhance engineering efficiency. This includes the development and open-sourcing of Velox, a unified execution engine that significantly improves efficiency and accelerates innovation in data management systems. The focus is on decomposing monolithic systems into modular components, utilizing open standards like Apache Arrow, and enabling domain-specific extensions.

All Meta trails
  1. Meta’s AI Storage Blueprint at ScaleThis post details the evolution of Meta's BLOB-storage architecture to specifically address the demands of AI workloads. It highlights the challenges of storage bottlenecks impacting GPU utilization and research velocity, and introduces a rebuilt foundation with a unified metadata schema, a fat client SDK eliminating dataplane proxy, and regional deployments. It also covers strategies for handling spikes and hot spots with distributed data caches and readplan metadata caches, and protocol optimizations like hedged reads and dynamic concurrency control.
  2. Migrating Data Ingestion Systems at Meta ScaleThis post details the successful large-scale migration of Meta's data ingestion system from a legacy architecture to a new, self-managed data warehouse service. Key contributions include the establishment of a three-phase migration lifecycle (Shadow Phase, Reverse Shadow Phase, Migration Cleanup) with defined success criteria (data quality, landing latency, resource utilization). It also describes the development of a custom data quality analysis tool using Scuba for real-time comparison of row counts and checksums, and strategies for handling rollout and rollback, including early signals via backfills and mechanisms to stop bad data propagation in CDC processes. The post highlights the use of automated tooling for monitoring and migrating tens of thousands of jobs in batches, considering factors like throughput and priority, and managing limited migration capacity.
  3. Introducing OpenZL: An Open Source Format-Aware Compression FrameworkThis post introduces OpenZL, an open-source format-aware compression framework designed for structured data. It aims to provide the performance of specialized compressors with the maintenance simplicity of a single binary. OpenZL achieves this by applying a configurable sequence of transforms to reveal patterns in structured data before compression, offering significant improvements in compression ratio and speed compared to generic compressors like Zstandard and xz, especially for structured datasets. It also introduces a trainer component for automatically generating optimal compression configurations and a universal decoder that remains consistent even as compression plans evolve.
  4. Creating AI agent solutions for warehouse data access and securityThis post details the development of an agentic system for data warehouse access, introducing data-user agents and data-owner agents to streamline data discovery, access requests, and security management. It describes how the data warehouse is represented as text-based resources for LLMs, and introduces context and intention management strategies. A deep dive into 'partial data preview' showcases the orchestration of user activities, user profiles, query shapes, data access budgets, and rule-based risk management to enable task-specific, context-aware data access.
  5. Accelerating GPU indexes in Faiss with NVIDIA cuVSThis post details the integration of NVIDIA cuVS into Meta's open-source Faiss library (v1.10), significantly accelerating GPU-based vector search. It highlights performance improvements in both inverted file (IVF) and graph indexing, showcasing substantial speedups in build times and reductions in search latency through collaboration with NVIDIA. This advancement extends Meta's capabilities in large-scale similarity search and AI infrastructure.
  7. How Meta understands data at scaleThis post details Meta's advancements in data understanding as part of the Privacy Aware Infrastructure (PAI) initiative. Key contributions include the development of a universal privacy taxonomy for standardized data privacy management, a "shift-left" approach integrating data schematization and annotations early in development, and the creation of OneCatalog for discovering, registering, and enumerating data assets. The post highlights solutions to challenges such as understanding data at scale (hundreds of data systems, millions of assets) through shared asset schema formats and managing long-tail systems; inconsistent definitions across diverse data systems by introducing a unified taxonomy of semantic types; missing annotations due to limited contextual information by combining schematization with in-code annotations and multiple classification signals; and organizational barriers through collaborative tooling and workflow integration. A walkthrough of understanding user data for the "Beliefs" feature in Facebook Dating illustrates the five-step approach: schematizing using Thrift IDL for a canonical DataSchema, predicting metadata at scale through the universal privacy taxonomy and data classification, and embedding privacy considerations throughout the data lifecycle.
  8. Data logs: The latest evolution in Meta’s access toolsThis post details the engineering challenges and solutions for providing users with access to granular 'data logs' from Meta's Hive data warehouse. It describes the infeasibility of direct per-user queries due to scale and partitioning, and the computational/storage costs of caching. The implemented solution involves batching user requests, using Core Workflow Service (CWS) for reliable task execution, and performing an INNER JOIN between user IDs and data rows within Dataswarm. Post-processing with PySpark splits data per user, and Hack language renders it into HTML. Key lessons learned include the critical importance of robust checkpointing for resilience against failures (e.g., out-of-memory errors during full table scans) and the implementation of verification steps (e.g., checking user ID columns) to ensure data correctness, particularly addressing Spark concurrency bugs that could lead to data being returned to the wrong user.
  9. How Meta discovers data flows via lineage at scaleThis post details Meta's implementation of data lineage for Privacy Aware Infrastructure (PAI), focusing on automated discovery of data flows across web, data warehouse, and AI systems. It introduces 'Privacy Probes' for runtime signal collection, comparing source and sink payloads to identify data matches with varying confidence levels (EXACT_MATCH, CONTAINS, NO_MATCH). The system uses static code analysis and runtime instrumentation to build lineage graphs, enabling scalable data flow discovery, efficient privacy control rollout (via Policy Zone Manager - PZM), and continuous compliance verification. The example of tracking 'religion' data in Facebook Dating illustrates the process of collecting signals from web endpoints, logging tables, databases, and propagating to data warehouses and AI systems.
  10. Inside Bento: Jupyter Notebooks at MetaThis post introduces Bento, Meta's internal distribution of Jupyter Notebooks, highlighting its use for prototyping and machine learning workflows. It details features built on top of Jupyter, including scheduled notebooks, sharing capabilities, and serverless notebooks leveraging WebAssembly in the browser. This expands the platform's flexibility and accessibility for data analysis and development.
  11. Serverless Jupyter Notebooks at MetaIntroduced serverless Jupyter notebooks for lite workloads within the Bento platform by integrating Pyodide and WebAssembly. This enables in-browser code execution, removing the need for remote compute provisioning for prototyping and simple ML workflows. Implemented a kernel bridge abstraction for seamless switching between server-based and serverless kernels, and adapted cell magics (e.g., %%sql, %%googlesheet, %%graphql) to run in the browser context. Utilized web workers to prevent browser lock-up during kernel operations and integrated with Tupperware for data uploads.
  12. Composable data management at MetaThis post details Meta's shift towards a composable data management architecture, moving away from fragmented monolithic systems. It introduces the concept of factoring out common logical components (language frontend, IR, query optimizer, execution engine, execution runtime) as reusable libraries. The core contribution is the development and open-sourcing of Velox, a unified, engine- and dialect-agnostic execution engine that leverages columnar and vectorized execution, compressed execution, lazy materialization, and adaptivity. The post also emphasizes the importance of open standards like Apache Arrow for interoperability and discusses the ongoing work in system-wide integration and extensibility for domain-specific features.
  13. Logarithm: A logging engine for AI training workflows and servicesIntroduced Logarithm, a new internal, serverless, multitenant logging engine designed for AI training workflows and services. Logarithm ingests and indexes over 100GB/s of logs in real-time, supporting thousands of queries per second with service-level guarantees on freshness, completeness, durability, and query latency. It supports unstructured text logs with optional metadata key-value pairs, including typed structures and regex-based extraction. The system architecture separates compute (ingestion and query) from storage, utilizes tiered storage (memory, local SSD, blob storage), and employs lightweight, disaggregated secondary indices with Bloom filters. Logarithm's design prioritizes scalability, fault tolerance, and efficient handling of high-throughput, time-ordered log data, with a focus on enabling debugging of complex AI training failures by collocating system and model telemetry.
  14. Aligning Velox and Apache Arrow: Towards composable data managementThis post details the convergence of Meta's Velox execution engine with Apache Arrow, resulting in three new format layouts in Apache Arrow 15.0.0: StringView, ListView, and Run-End-Encoding (REE). StringView offers improved inlining for short strings, faster comparisons, and flexible data layout. ListView enables out-of-order writes for variable-sized containers, facilitating efficient conditional execution. REE adds support for constant and run-length-encoded data, common in data warehousing. This convergence aims to create a more unified and composable data landscape.
  15. Automating data removalThis post introduces the subsystem of SCARF responsible for automating the identification and safe removal of unused data types. It details how SCARF scans production data systems to identify unused tables or assets, measuring both static code usage (via Glean) and runtime access patterns. The system orchestrates removal by blocking reads/writes after a notification period and then deleting the data. It also addresses cross-system dependencies and code usage complexities to ensure thorough and safe data cleanup at scale, having removed petabytes of unused data and millions of data types.
  16. Scheduling Jupyter Notebooks at MetaThis post details the development of a scheduled notebooks infrastructure for Meta's Bento Jupyter notebooks platform, integrating it with the Dataswarm batch ETL pipeline framework. Key technical contributions include the BentoOperator, which executes notebooks in network-isolated containers, uses custom cells for structured data fetching (e.g., SQL cells) and writing (e.g., pandas.read_csv/to_csv), and leverages input/output data annotations for lineage tracking. The system enforces privacy through integration with Meta's data purpose framework and a "purpose policy zone." The workflow involves UI for schedule setup, diff creation for review, and a debugging interface. Future work aims to expand supported notebook types, introduce parameterized notebooks, and implement event-based scheduling.
  17. Building and deploying MySQL Raft at MetaThis post details the implementation of MySQL Raft at Meta, a system that integrates the Raft consensus engine with MySQL to create a replicated state machine. It replaces the previous semisynchronous replication protocol with a Raft-based approach, enhancing reliability, safety, failover time, and operational simplicity. Key technical contributions include the development of the kuduraft library (a fork of Apache Kudu with enhancements like FlexiRaft, proxying, compression, log abstraction, and primary ban), the MyRaft MySQL plugin for interfacing with Raft, and novel replication topologies and write transaction flows leveraging Raft for consensus. The post also describes modifications to MySQL's binary log and crash recovery mechanisms to support the Raft protocol.
  18. Introducing Velox: An open source unified execution engineIntroduced Velox, an open-source unified execution engine designed to accelerate and streamline the development of data management systems. Velox consolidates common data-intensive components of computation engines, offering a framework for consistent semantics and reducing work duplication. It integrates with existing systems like Presto (Prestissimo), Spark (Gluten), and PyTorch (TorchArrow), demonstrating significant performance improvements (up to 10x speedups) in various benchmarks and real-world workloads. Velox provides a generic type system, Arrow-compatible vector layout, vectorized expression evaluation, and common SQL operators, along with I/O and resource management primitives.
  19. Tulip: Modernizing Meta’s data platformThis post details the engineering, scaling, and non-technical challenges of migrating Meta's exabyte-scale data platform to the new Tulip serialization format for analytics logging. It describes the motivation for the migration, driven by reliability and efficiency issues with legacy (de)serialization formats. Key technical solutions include a mixed-mode wire format to support gradual migration, a 'Reader' library to abstract serialization from consumers, and format conversion in consumers for infeasible updates. Debugging tools like 'loggertail' and 'shadow loggers' were developed for validation and end-to-end testing. The migration was executed in two phases: planning/preparation/experimentation and scaling, with a focus on tooling, automation, and managing non-technical aspects.
  20. Enabling static analysis of SQL queries at MetaThis post introduces UPM (Unified Programming Model), a standalone library for static analysis of SQL code and enhancement of SQL authoring. UPM represents SQL queries as semantic trees, enabling tools to perform static analysis (linters, data lineage), query rewriting, or act as a pluggable SQL front-end for query engines. It addresses challenges with embedded query engine analysis logic and limited type systems by providing a unified SQL dialect and enhanced type-checking using user-defined types. UPM-powered tools can now build column-level data lineage graphs across the warehouse.
  21. Tulip: Schematizing Meta’s data platformIntroduced Tulip, a new binary serialization protocol designed to replace legacy formats like Hive Text Delimited and JSON for data logging and analytics at Meta. Tulip addresses standardization, reliability, efficiency, correctness, and forward/backward compatibility issues by using TCompactProtocol and a robust schema management system. It achieves significant performance gains, using 40-85% fewer bytes and 50-90% fewer CPU cycles for serialization/deserialization compared to previous formats. The system assigns field IDs automatically and ensures indelible field history for complete backward compatibility.
  22. Open-sourcing TAOBench: An end-to-end social network benchmarkOpen-sourced TAOBench, a benchmark framework and workload configurations designed to accurately model social network graph workloads. TAOBench captures transactional requirements, data colocation preferences, request distributions, and multitenant behavior by profiling requests from Meta's TAO graph data store. The benchmark demonstrates its utility by reporting results for five popular distributed databases and highlighting its use cases for testing new features, optimizations, and speculative workloads.
  23. Introducing Zelos: A ZooKeeper API leveraging DelosIntroduced Zelos, a ZooKeeper API built on the Delos distributed system platform. Zelos aims to replace Meta's existing ZooKeeper clusters by providing a more modular, extensible, and performant variant. Key technical contributions include addressing the impedance mismatch between ZooKeeper's session management and Delos's linearizable log by implementing speculative execution for session ordering and a RequestProcessor for ordering reads within sessions. It also details the challenges and solutions for transparently migrating ZooKeeper-based applications to the new Zelos architecture.
  24. Delta: A highly available, strongly consistent storage service using chain replicationIntroduced Delta, a new object storage service designed for critical build and distribution artifacts, prioritizing high availability and strong consistency. Delta leverages chain replication for data redundancy and fault tolerance, trading latency and storage efficiency for simplicity and reliability. It supports basic operations (put, get, delete, list) and is horizontally scalable. The post details the architecture of Delta, its comparison with quorum replication, and its failure and recovery modes, including host failure detection and recovery mechanisms.
  25. SQL Notebooks: Combining the power of Jupyter and SQL editors for data analyticsIntroduced SQL Notebooks, a tool that merges SQL IDEs and Jupyter Notebooks for data analytics. It enables modular SQL queries using CTE-like cell referencing, sandboxed Python for post-processing and visualizations, and secure, scheduled output sharing by enforcing ACLs on SQL queries. This addresses scalability and security limitations of traditional notebooks for big data analytics.
  26. Augmenting Flexible Paxos in LogDevice to improve read availabilityThis post details improvements to LogDevice, Meta's scalable distributed log storage system, by augmenting its Flexible Paxos consensus algorithm to enhance read availability. It addresses the trade-offs in Flexible Paxos concerning read and write availability, particularly in the context of failures and maintenance. The post explains how LogDevice's recovery process and a separate membership mechanism using ZooKeeper were adapted to maintain write availability while improving read availability, even in scenarios where a majority of acceptors are unavailable, by allowing reads without a strict quorum and addressing 'stuck recovery' issues.
  27. RAMP-TAO: Layering atomic transactions on Facebook’s online graph storeIntroduced RAMP-TAO, a new protocol that layers atomic transaction guarantees on top of TAO, Facebook's online social graph store. RAMP-TAO addresses the challenge of providing transactional semantics over an eventually consistent, massive-scale data store by focusing on guarding against fractured reads for recently updated data, minimizing performance and storage overheads for existing non-transactional workloads. It achieves over 99.9 percent of reads completing in one round-trip to the local cache with tail latency on par with existing TAO reads, incurring overhead only for opt-in applications.
  28. How we built a general purpose key value store for Facebook with ZippyDBThis post introduces ZippyDB, a large-scale, strongly consistent, geographically distributed key-value store at Facebook, built on RocksDB. It details ZippyDB's history, architecture (tiers, regions, shards, μshards, Akkio), data model (key-value, TTL, transactions, conditional writes), and configurable consistency/durability levels. ZippyDB addresses the need for a unified, scalable key-value store by reusing existing infrastructure like Data Shuttle, Shard Manager, and ZooKeeper, thereby reducing duplicated efforts across teams and enabling faster product development.
  29. Migrating Facebook to MySQL 8.0This post details the technical challenges and solutions involved in migrating Facebook's core workloads from MySQL 5.6 to MySQL 8.0. It covers porting over 1,700 custom code patches, ensuring replication compatibility, minimizing application query changes, and fixing performance regressions. The migration involved a phased approach for replica sets, standardization on row-based replication, and extensive automation validation for both infrastructure and application queries. Specific issues encountered included binlog event format incompatibilities, conflicting error codes, reliance on .frm files, and dynamic privilege support. Application-level issues like reserved keyword conflicts and REGEXP incompatibilities were also addressed.
  30. Ribbon filter: Practically smaller than Bloom and XorIntroduced the Ribbon filter, a new space-efficient data structure that is practically smaller than Bloom filters. Ribbon filters offer near-continuous configurability in the accuracy-versus-space trade-off, with space overhead as low as 1 percent or less, and O(1) query times. They save roughly 1/3 of memory compared with Bloom filters by solving a linear system using Gaussian elimination on a band matrix structure, with an on-the-fly construction algorithm called Rapid Incremental Boolean Banding ON the fly (Ribbon).
  31. Consolidating Facebook storage infrastructure with Tectonic file systemIntroduced Tectonic, a data center scale distributed file system designed to consolidate storage infrastructure. Tectonic supports exabyte-scale, isolates performance between tenants by managing resources between traffic groups, and enables tenant-specific optimizations through a client-driven microservice architecture. This consolidation has reduced data warehouse clusters by 10x, improving resource utilization and operational simplicity.
  32. Meet Kats — a one-stop shop for time series analysisIntroduced Kats, a new comprehensive Python library for generic time series analysis. Kats provides algorithms and models for forecasting (10+ models, ensembling, meta-learning, backtesting, hyperparameter tuning, prediction intervals), detection (seasonality, outlier, change point, slow trend changes), feature extraction (65 statistical features), embedding, and multivariate analysis. It also includes utilities like time series simulators. Kats aims to standardize and connect various domains within time series analysis, empowering users to explore data characteristics, predict values, monitor anomalies, and integrate these insights into ML models and pipelines.
  33. Nemo: Data discovery at FacebookThis post introduces Nemo, an internal data discovery engine at Facebook. Nemo revamps the original data discovery solution by using Unicorn search infrastructure for scalability and more sophisticated search capabilities, including natural language queries. It incorporates personalization signals and a gradient-boosted decision tree model for post-ranking. The architecture includes bulk and instant indexing, and the serving component uses spaCy for NLP, Unicorn for retrieval and initial ranking, and an FBLearner-trained ML model for post-processing. Nemo also uses trust signals to rank results, improving data quality and recency. Lessons learned include the challenges of A/B testing and labeling in a large organization, and the importance of Nemo in maintaining data health.
  34. CG/SQL: Easy, accurate code generation for SQLiteIntroduced CG/SQL, a code generation system for SQLite that compiles stored procedures written in a T-SQL variant into C code using SQLite's C API. This system provides compile-time error checking, automatic schema management and upgrades, test code generation, and query plan analysis, aiming to simplify the creation of complex, maintainable data access layers for SQLite.
  35. Private matching for compute: New solutions to the problem of enabling compute on private set intersectionsThis post introduces two new cryptographic protocols, Private-ID and PS3I, for private matching on set intersections. These protocols enable secure data alignment and analysis between different parties without revealing sensitive information, extending the data warehousing and analytics platform's capabilities to handle privacy-sensitive computations and large-scale data joins.
  36. Turbine: Facebook’s service management platform for stream processingIntroduced Turbine, a scalable service management platform for Facebook's stream processing service. Turbine bridges the gap between general-purpose cluster management frameworks and stream processing requirements, featuring a fast task scheduler, an efficient predictive autoscaler, and a fault-tolerant update mechanism. It manages tens of thousands of machines and thousands of streaming pipelines, processing hundreds of gigabytes of data per second in real-time, balancing workload fluctuations, predicting load spikes, and completing high-scale updates efficiently.
  37. ZooKeeper Meetup@Facebook: Advancing the state of distributed coordinationThis post details advancements in ZooKeeper at Facebook, including a 10x improvement in write throughput via snapshot scheduler changes and batch commits. It also covers optimizing semaphores with a new lock API to reduce server round trips and QPS. Scalability improvements include demand control for multitenancy (data size and request rate quotas) to mitigate noisy neighbor issues, and zero-downtime migration strategies. Furthermore, the entire ZooKeeper fleet has been containerized, moving away from bare-metal deployments, simplifying fleet management and deployments.
  38. Scribe: Transporting petabytes per hour via a distributed, buffered queueing systemThis post details the architecture and evolution of Scribe, a distributed, buffered queueing system designed to transport petabytes of logs per hour from millions of machines. It covers the system's design decisions, including simplicity, availability, scalability, and multitenancy, and explains its components like Scribed (local daemon), Write Service, and its reliance on LogDevice for durable storage, replacing HDFS. The post also discusses the trade-offs in log delivery guarantees and the evolution from Scribe's first incarnation to its current, simplified production version.
  39. MaRS: How Facebook keeps maps current and accurateIntroduced MaRS (Machine-Augmented Automatic Review System) and Logical Changesets (LoChas) to efficiently ingest and maintain up-to-date OpenStreetMap (OSM) data for Meta's applications. LoChas break down OSM changesets into individual CRUD operations and cluster them for efficient human review. MaRS uses heuristics and ML to automate the evaluation of LoChas, reducing the workload requiring human intervention. This system ensures data freshness and correctness by processing changes in an idempotent, commutative, and independent manner, allowing for parallel processing and binary accept/reject decisions.
  40. Redesigning our systems to provide more control over Off-Facebook activityThis post details the technical redesign of Meta's systems to provide users with control over Off-Facebook activity. It describes the challenges of storing and processing this data, which was primarily in the data warehouse, and the need to move away from time-based partitioning to user-centric organization for this specific feature. Key technical solutions include the introduction of Separable Identifiers (SIDs) to decouple user IDs (UIDs) from off-site activity data in the data warehouse, the use of Encrypted IDs (EIDs) for compound data types, and Measurement IDs (MIDs) to enable aggregated measurement reporting while respecting user privacy. The post also outlines strategies for handling special cases, such as ensuring data security and safety investigations, and managing Facebook Login integrations.
  41. OIL+VCache: File abstraction for distributed systemsIntroduced OIL (Output-Input Language) as a unified namespace and abstraction for files across heterogeneous storage systems, allowing configuration of I/O trade-offs via Directed Acyclic Graphs (DAGs) instead of coding. Coupled OIL with VCache, a distributed virtual memory system, to provide significant efficiency and performance gains. OIL extends POSIX APIs to better capture distributed system nuances, enabling per-file or per-I/O trade-off adjustments and allowing files to retain the same name even when migrating storage systems. VCache integrates with OIL to provide layered caching, supporting various write modes and delegation for buffering and durability. The system uses DAGs for both data and metadata, with the metadata DAG handling mutual exclusion and locking, and the data DAG managing byte-stream operations. OIL+VCache has been deployed to enhance live video viewer systems, improving reliability and latency while reducing computational overhead.
  42. Aria Presto: Making table scan more efficientThis post introduces optimizations for Aria Presto's table scan efficiency, specifically for ORC format data. Key contributions include: 1. Subfield pruning: Extracting only referenced complex type subfields (e.g., specific map keys) instead of entire complex types. 2. Adaptive filter ordering: Dynamically reordering filter evaluation to prioritize filters that drop more rows with fewer CPU cycles. 3. Efficient row skipping: Reading and discarding only necessary values for rows that pass filters, rather than reading entire rows and then discarding them. These optimizations are implemented by shifting filter evaluation into the Hive connector and introducing a stream reader responsible for applying simple filters and producing filtered row number arrays. A record reader coordinates stream readers, tracks filter efficiency, and handles multi-column filter functions. A working prototype shows ~20% CPU gains on production workloads.
  43. Delos: Simple, flexible storage for the Facebook control planeIntroduced Delos, a new architecture for replicated storage systems designed for control plane services. Delos features a modular, layered design separating API materialization, update ordering (VirtualLog/SimpleLogs), and reconfiguration. This allows for fast deployment (8 months for initial version), rapid evolution (10x latency reduction via hot-swapping ordering mechanisms), and flexibility in API and performance/reliability guarantees. It uses a VirtualLog abstraction that chains SimpleLogs, enabling dynamic switching between different ordering mechanisms (e.g., ZooKeeper-based, native converged, disaggregated LogDevice) based on latency SLAs.
  44. Open-sourcing F14 for faster, more memory-efficient hash tablesIntroduced F14, a 14-way probing hash table implementation within the Folly C++ library, designed for improved performance and memory efficiency over previous specialized hash table implementations. F14 utilizes vector instructions (SSE2/NEON) for parallel intra-chunk searching and a novel reference-counted tombstone strategy (overflow bits) to handle collisions and erasures efficiently. It offers multiple memory layouts (F14FastMap/Set, F14NodeMap/Set, F14Vector) to cater to different use cases, particularly optimizing for scenarios without long-lived references to entries.
  45. Open-sourcing homomorphic hashing to secure update propagationIntroduced LtHash, a homomorphic hashing algorithm based on lattice cryptography, for secure and scalable update propagation in configuration management systems. This addresses the challenges of maintaining data integrity across a large network of peers by enabling efficient recomputation of database hashes after updates, without iterating over the entire database. The implementation is open-sourced in Folly.
  46. 5 ways Facebook improved compression at scale with ZstandardThis post details the integration and benefits of Zstandard (zstd) compression at Facebook, focusing on five key areas: large payload backups and package distribution (introducing multithreading, long range mode, and automatic level determination), data warehousing (increasing block size and reducing data transformations), compressed filesystems (SquashFS and Btrfs for faster deployments and tighter storage), and databases (hybrid compression with RocksDB). These enhancements have led to significant improvements in compression ratios, decompression speeds, and reduced resource consumption across these systems.
  47. HyperLogLog in Presto: A significantly faster way to handle cardinality estimationThis post details the implementation of the HyperLogLog (HLL) algorithm in Presto to significantly improve the performance of cardinality estimation for large datasets. It explains the evolution of cardinality estimation algorithms from simple minimum value estimation to probabilistic counting and LogLog, culminating in HLL. The post highlights the memory efficiency and speed improvements (7x to 1,000x) achieved by using HLL in Presto for tasks like analyzing A/B tests.
  48. Data @Scale – Boston recapThis post summarizes presentations from the Data @Scale conference in Boston, highlighting advancements in data systems. Key contributions include techniques for leveraging sampling to reduce data warehouse resource consumption at Facebook, improving approximate query results and offsetting compute/storage demand. It also covers the evolution of Apache Cassandra with Transient Replication and Cheap Quorums for disk space and compute savings, and Facebook's Deletion Framework for managing data deletion across distributed systems. Other contributions include Wayfair's transformation of data plumbing with Tremor, HubSpot's migration of Elasticsearch instances for GDPR compliance, and the development of Presto's Cost-Based Optimizer (CBO) for performance improvements in join operations and statistics collection.
  49. Managing data store locality at scale with AkkioIntroduced Akkio, a new data placement service (DPS) designed to optimize data retrieval speed and reduce storage and network costs in geo-distributed data centers. Akkio operates at a microshard granularity, determining optimal data placement based on access patterns and available capacity. It addresses the limitations of traditional caching for Facebook's workloads by handling strong consistency requirements and high write volumes more effectively. The system leverages ZippyDB for storing access patterns and metadata, and serializes migrations through locks to prevent concurrent operations.
  50. Open-sourcing LogDevice: A distributed data store for sequential dataThis post announces the open-sourcing of LogDevice, a distributed data store for sequential data developed and deployed within Facebook. LogDevice provides durability, high availability, and total order of records under failures, making it suitable for workloads like event streaming, replication pipelines, transaction logs, and deferred work journals. It also introduces LDShell, a command-line administration tool.
  51. Location-Aware Distribution: Configuring servers at scaleIntroduced Location-Aware Distribution (LAD), a new peer-to-peer system for distributing configuration changes to millions of servers. LAD improves upon the previous ZooKeeper-based system by supporting larger updates (100 MB vs 5 MB), scaling to 40,000 subscribers per distributor (vs 2,500), and reducing distribution latency. LAD separates data storage from distribution, utilizes a tree-based peer-to-peer network for efficient updates, and employs sharding to manage metadata distribution. Lessons learned emphasize the critical role of tooling, monitoring, failure injection testing, and continuous testing for operating and maintaining large-scale P2P systems.
  52. Migrating Messenger storage to optimize performanceThis post details the migration of Messenger's storage system from HBase to MyRocks, integrated with RocksDB as a MySQL storage engine, and leveraging Facebook's Lightning Server SKU with NVMe flash storage. It describes the challenges of migrating petabytes of data for over a billion users without disruption, including redesigning the data schema, ensuring data consistency through dual-writing and validation, and implementing two distinct migration flows (normal and buffered) to handle different account types and traffic loads. The migration resulted in a 90% reduction in storage consumption, 50x lower read latency, and improved system resiliency and disaster recovery capabilities.
  53. Migrating a database from InnoDB to MyRocksThis post details the successful migration of Facebook's UDB tier from the InnoDB storage engine to MyRocks, a RocksDB-based engine for MySQL. The migration achieved a 50% reduction in storage usage by addressing the space-bound nature of InnoDB on flash storage. Key technical contributions include the development and application of comprehensive data consistency verification tools (primary/secondary key checks, cross-instance checks, shadow query replay), a phased migration strategy leveraging MyRocks' ability to replicate from InnoDB and vice-versa, and addressing technical challenges like buffered I/O and slow file deletion on flash storage. The post also highlights lessons learned regarding migration time estimation, understanding service usage for feature prioritization, starting small with new software deployments, and the benefits of open-source collaboration.
  54. LogDevice: a distributed data store for logsIntroduced LogDevice, a distributed data store specifically designed for logs. LogDevice provides highly available and durable record storage with repeatable total ordering, addressing challenges in data pipelines, indexing, and distributed transactions at Facebook's scale. Key features include record-oriented append-only storage with trimming, non-deterministic record placement for improved write availability and load balancing, and a sequencer-based architecture that decouples record ordering from storage.
  55. Data @Scale 2017 RecapThis post summarizes presentations from the Data @Scale 2017 conference, detailing advancements in several key data engineering areas. It covers the evolution of globally-distributed databases (Microsoft Azure), interactive analytics DBMS like ClickHouse (Yandex), machine learning serving platforms (Pinterest), micro-service architectures beyond request/reply with Cadence (Uber), platforms for metrics computation and experimentation (LinkedIn's UMP and XLNT), the evolution of globally distributed data management systems like Spanner (Google), and Facebook's system for bulk data movement across global storage. It also touches upon accelerating machine learning with infrastructure design and the use of programmable hardware in cloud specialization.
  56. OnlineSchemaChange rebuilt in PythonThis post announces the rebuilding of the OnlineSchemaChange (OSC) tool in Python, replacing the original PHP version. The new Python version offers improved ease of use with a standalone CLI, better testability through a more flexible code structure and integration testing framework, and enhanced reliability with data consistency checks. Future plans include triggerless schema changes and utilizing native online DDL.
  57. The software ecosystem for storage hardwareThis post details the validation of various software-defined storage solutions (NetApp ONTAP Select, IBM Spectrum Scale, Excelero NVMe over Fabrics, Weka.IO, Hedvig, Red Hat GlusterFS, and Ceph) on open hardware platforms like Leopard servers and Open Vault JBODs within the Disaggregate Lab. It highlights the growing software ecosystem for open hardware and its appeal to companies of all sizes.
  58. Using Apache Spark for large-scale language model trainingThis post details the migration of a large-scale, complex (100+ stage) N-gram language model training pipeline from HQL over Hive to Apache Spark. It highlights the use of Spark's imperative DSL and RDD interface to gain finer control over data distribution, avoid data skew, and implement application-specific optimizations. Key challenges addressed include managing large '0-shards' for smoothing, potential job instability due to data skew, and executor/shuffle service OOM errors. The solution involved a progressive sharding approach, iteratively refining data distribution based on N-gram history length (1-word, 2-word, etc.) and dynamically adjusting shard sizes to ensure balanced workloads. The use of Spark's `pipe()` operator facilitated the integration of existing C++ binaries, enabling a smooth transition without re-implementing core logic.
  59. Beringei: A high-performance time series storage engineThis post details the development and open-sourcing of Beringei, a high-performance in-memory time series storage engine designed to address the scaling limitations of HBase for Facebook's monitoring infrastructure. It introduces a lossless streaming compression algorithm that achieves over 90% compression for time series data by using delta-of-delta encoding for timestamps and XOR-based compression for values. Beringei offers very fast write rates and low read latencies, storing up to 10 billion unique time series and serving 18 million queries per minute, powering Facebook's real-time performance and health monitoring.
  60. Continuous MySQL backup validation: Restoring backupsThis post details the ORC (ORC Restore Coordinator) system, a component of Meta's continuous MySQL backup validation infrastructure. It describes the architecture of ORC, including the Warchief (load balancer), ORC DB (state management), and Peons (restore workers). The post elaborates on the five stages of a restore job within a peon: SELECT, DOWNLOAD, LOAD, VERIFY, and REPLAY. A significant technical contribution is the detailed explanation of the binlog selection logic, which leverages Global Transaction Identifiers (GTIDs) to accurately determine and replay transactions for point-in-time restores, overcoming challenges with primary/secondary backup sources and ensuring data integrity.
  61. A comparison of state-of-the-art graph processing systemsThis post provides a quantitative and qualitative comparison between Apache Giraph and Spark GraphX for large-scale graph processing at Facebook. It details the methodology, experiment setup, and key findings regarding performance, scalability, and usability. The study reveals that Giraph can process significantly larger graphs and performs better and more efficiently than GraphX for production workloads, while GraphX offers some advantages in ease of development for certain preprocessing tasks.
  62. Facebook announces new tech at @Scale 2016Introduced Zstandard, a new compression algorithm that offers smaller and faster compression, twice the decompression speed, and higher compression ratios, leading to faster data transfer and smaller storage requirements. Announced MyRocks, an open-source MySQL storage engine built on RocksDB, which reduces storage requirements by 50% in production user database tiers. Developed a 360 video stabilization technology using a combination of 2D and 3D stabilization methods with a new 'deformed rotation' motion model, improving video quality, reducing bit rate, and enabling fast upload and playback. Presented BrowserLab, an automated testing environment for detecting regressions as small as 20ms, handling dynamic content, isolating server variance, and accounting for non-deterministic code. Discussed Infer, an open-source static analyzer for Java, C, and Objective-C, with ongoing efforts to add support for C++.
  63. Apache Spark @Scale: A 60 TB+ production use caseThis post details the migration of a large-scale Hive-based feature preparation pipeline for entity ranking to Apache Spark. It highlights the challenges of the previous sharded Hive implementation (3-day runtime, difficult monitoring) and the technical solutions implemented to run a single Spark job on 60 TB of compressed data with a 90 TB shuffle. Key contributions include reliability fixes for node reboots (PipedRDD robustness, configurable fetch failures, restartable shuffle service), addressing driver issues (O(N^2) operations, excessive speculation), and executor OOM fixes. Performance optimizations involved memory leak fixes in the sorter and Snappy optimizations. The post also details the tools used for performance bottleneck identification (Spark UI Metrics, Jstack, Perf/Flame Graph) and emphasizes the contribution of these improvements back to the open-source Apache Spark project.
  64. MyRocks: A space- and write-optimized MySQL databaseIntroduced MyRocks, a new open-source MySQL storage engine that integrates RocksDB as a backend. MyRocks offers significant improvements in space efficiency (50% less storage compared to compressed InnoDB) and reduced write amplification by leveraging RocksDB's LSM-tree structure, append-only nature, and prefix key encoding. Benchmarks show MyRocks is two times smaller than compressed InnoDB and writes orders of magnitude less. The post also details the migration process from InnoDB to MyRocks in production without downtime.
  65. Data @Scale, June 2016 — RecapThis post is a recap of the Data @Scale conference, highlighting talks on exabyte storage systems (Dropbox), analytics for storage systems (Qumulo), MPP Shared-Nothing Database on Flash (Facebook's Presto Raptor), lessons from studying modern databases and key-value stores (University of Wisconsin-Madison), flexible consistency levels with Manhattan (Twitter), scale-out resource management with YARN++ (Microsoft), Apache Beam for unified batch and streaming data processing (Google), data integration for visual analysis (Tableau), and disaggregated storage and compute for Map/Reduce (Facebook).
  66. Dragon: A distributed graph query engineIntroduced Dragon, a distributed graph query engine, to address the challenges of high-volume, multi-hop queries on Facebook's social graph. Dragon builds upon TAO by creating various indices to improve the efficiency of fetching, filtering, and reordering data, reducing network transfer and improving latency. It leverages techniques like socially aware inverted indices and optimized storage layouts based on query patterns, and supports functional programming primitives for complex filtering and ordering. Dragon aims to serve queries from cache with high availability and consistency, allowing applications to focus on business logic.
  67. Improving software RAID with a write-ahead logThis post introduces the implementation of a write-ahead log for software RAID 5/6 to address the 'write hole' problem, improve rebuild performance, and optimize write performance. The solution involves writing data and parity to a log on an SSD before persistent writes to RAID disks, ensuring consistency after power failures and enabling aggregation of writes to avoid read-modify-write penalties. The implementation details include metadata structures, checksum mechanisms, and reclaim strategies. Performance testing with different disk configurations is presented, along with future work on NVRAM/NVDIMM support and further write performance improvements.
  68. Building the Presto communityThis post details the creation and open-sourcing of Presto, a distributed SQL engine designed for interactive analytic queries against large datasets. It highlights Presto's architecture, its significant query performance improvements over traditional MapReduce jobs, and its adoption by various companies like Airbnb, Dropbox, and Netflix. The post also discusses the growth of the Presto community, with an increase in contributors and support from cloud platforms and enterprise vendors.
  69. Improving Facebook’s performance on Android with FlatBuffersThis post introduces the adoption of FlatBuffers as a data storage format for Facebook's Android application. It details the performance issues encountered with JSON parsing, such as slow parsing speeds, parser initialization delays, and significant garbage collection pressure. The post explains how FlatBuffers, with its ability to directly access data without full deserialization, addresses these issues. It also describes custom mutation extensions built on FlatBuffers to handle data modifications and the resulting 'Flat Models' architecture that simplifies the integration between UI and storage layers. The post quantifies the performance improvements achieved, including reduced story load times, decreased memory allocations, faster cold starts, and smaller storage size.
  70. Inside Data@Scale 2015This post summarizes presentations from the Data@Scale 2015 event, detailing technical advancements in distributed data storage and processing. Key contributions include: Eric Hwang's talk on Presto, an open-source distributed SQL query engine for ad-hoc analysis; Sundaram Narayanan's presentation on LogLens, Twitter's real-time log indexing and search service; Samir Goel's discussion of Firefly, Dropbox's index and search system for managing billions of daily file changes; Dharma Shukla's overview of Azure DocumentDB, a cloud-based NoSQL database; Muthu Annamalai's explanation of ZippyDB, Facebook's key/value memory cache service; Kestutis Patiejunas's talk on Facebook's multi-exabyte cold storage systems (HDD and Blu-ray based); and Brian Beach's presentation on Backblaze Vaults, a distributed storage system.
  71. Recommending items to more than a billion peopleThis post details the development of a distributed algorithm for collaborative filtering (CF) to handle Facebook's massive datasets (100 billion ratings, over a billion users, millions of items). It addresses the limitations of standard distributed matrix factorization approaches by introducing a rotational hybrid approach that leverages worker-to-worker messaging in Apache Giraph. This new approach significantly reduces network traffic and eliminates skewed item degree problems, improving scalability for recommendation systems.
  72. How RocksDB is used in osqueryThis post details how osquery utilizes RocksDB as its embedded persistent key-value store. It explains osquery's functionality as an OS instrumentation framework and how RocksDB enables fast, persistent data storage and access. The article highlights the event-based pub/sub system within osquery, demonstrating how RocksDB is used to store and compare query results over time to detect state changes. It also introduces the concept of event publishers and subscribers, using udev events as a concrete example of how osquery reacts to OS-level events efficiently by leveraging RocksDB for state management.
  73. Parse now running MongoDB on RocksDBThis post details Parse's adoption of RocksDB as a storage engine for MongoDB in production. It highlights significant performance gains, particularly in write operations (50x faster inserts) and data compression (70%-90%), which allows for more indexes to fit into RAM, leading to faster overall performance. This integration enhances MongoDB's existing strengths in failover and workload flexibility.
  74. Even faster: Data at the speed of Presto ORCThis post details the development and integration of a new ORC reader for Presto, designed to significantly improve data processing performance. Key contributions include enabling columnar reads, implementing predicate pushdown by leveraging ORC's min/max statistics, and introducing lazy reads to optimize data access based on query predicates. The new reader supports both ORC and DWRF formats and demonstrates substantial speedups in raw decoding and end-to-end query latency compared to previous implementations and other data formats like RCFile and Parquet.
  75. Alibaba joins WebScaleSQLAlibaba's MySQL team has joined the WebScaleSQL project, contributing bug fixes, performance improvements, enhanced large-scale parallelism, and micro-optimizations in compression. This collaboration is expected to accelerate progress on planned replication and GTID server-side fixes, async client features, logical read-ahead, query throttling, improved server-level thread pooling, large InnoDB page support, and memory usage reduction.
  76. Data @Scale – Boston RecapThis post details presentations from the Data @Scale – Boston conference, highlighting advancements in large-scale data processing and systems engineering. Key technical contributions include: Twitter's use of the Lambda architecture and probabilistic algorithms for real-time high-volume analytics processing; Facebook's technical challenges and infrastructure enablers for the 'Lookback videos' project, focusing on compute, network, storage, distribution, projections, and modeling; Vertica's Database Designer (DBD) for automated projection design in columnar databases and Live Aggregate Projections (LAPs) for faster querying through pre-aggregation; Wayfair's scaling of Redis and Memcached using composable tools like Ketama and Twemproxy for a resilient distributed caching system; TripAdvisor's lessons learned in handling operations at scale with Hadoop, log shipping, and anomaly detection; Facebook's implementation of spatial indexing on RocksDB for efficient geo-spatial data storage and optimization for various workloads; Facebook's use of graph partitioning for optimizing distributed systems serving graph-based datasets; Constant Contact's scaling strategies using Cassandra for key-value data and sharded MySQL for relational data, and their analytics platform leveraging Hadoop; and VoltDB's architecture for high-volume transactional and stream processing.
  77. Audience Insights query engine: In-memory integer store for social analyticsIntroduced the Audience Insights (AI) query engine, a system designed for rapid processing of social analytics data. Key technical contributions include a hybrid in-memory/flash disk integer store, a fan-out distributed architecture with aggregation and leaf tiers, sharding by user with 1024 shards, and the use of bitset-based inverted indices for efficient filtering (replacing user IDs with bitmasks for faster bitwise AND operations). Implemented cumulative bitmap indices for fast range filters using XOR. Addressed memory constraints by using bitmaps for dense attributes and traditional indexing for sparse attributes, converting sparse indices to bitmaps at query time. Optimized group-by operations using bitset-based filtering and converted 64-bit integer entity IDs to 32-bit vector indices. Improved cache efficiency by organizing user attributes in a columnar-like structure. Scaled affinity computation using greedy selection over locally computed candidates at each shard. Leveraged GPU hardware for offloading computationally intensive operations like affinity computation. Implemented tail latency reduction techniques by cutting execution for high-latency nodes. The engine is implemented in C++ and uses internal Facebook frameworks.
  78. @Scale 2014: Recap of Data TrackThis post details advancements in real-time analytics pipelines for mobile applications at Facebook, the evolution of Netflix's data platform centered around S3 and stream processing with Mantis, Pinterest's graph storage service (Zen), and Facebook's open-sourcing of mcrouter for scaling memcached deployments. It also touches upon Box's challenges in building scalable enterprise cloud infrastructure with Tron and Credence, Facebook's interactive A/B analysis tool Deltoid, Facebook's mobile-first backend for Messenger using MySQL, and YouTube's use of Vitess for scaling MySQL.
  79. Lessons from Deploying MySQL GTID at ScaleThis post details the deployment of MySQL's Global Transaction ID (GTID) feature across Facebook's production MySQL instances. It highlights the benefits of GTID for failover, point-in-time backup recovery, and hierarchical replication. The post outlines significant challenges encountered during deployment, particularly with existing high-availability strategies, and describes custom solutions developed in fb-mysql to overcome these. These include relaxing gtid_mode constraints during deployment, fixing bugs related to binary log scanning and crash-safety with less durable settings, and enabling multi-threaded secondary replication with relay_log_recovery. It also covers preparation steps like auditing applications for GTID-incompatible statements and developing helper scripts for statement skipping. The work done is planned to be open-sourced via WebScaleSQL.
  80. Introducing mcrouter: A memcached protocol router for scaling memcached deploymentsThis post introduces mcrouter, a memcached protocol router developed by Meta to scale memcached deployments. It highlights mcrouter's ability to handle massive request volumes, its open-source release, and its features like connection pooling, multiple hashing schemes, prefix routing, replicated pools, production traffic shadowing, online reconfiguration, flexible routing, health monitoring, cold cache warmup, broadcast operations, reliable delete streams, multi-cluster support, rich stats, quality of service controls, large value handling, multi-level caches, IPv6 and SSL support, and a multi-threaded architecture. This directly addresses the need for efficient and scalable data caching as a critical component of Meta's data infrastructure.
  81. HydraBase – The evolution of HBase@FacebookIntroduced HydraBase, a system that enhances HBase reliability by using a quorum-based approach with RAFT consensus for region hosting. This allows for faster failover and higher availability compared to traditional HBase primary-secondary setups, by having multiple region servers synchronously write to the WAL and tolerating up to F failures in a 2F+1 quorum. HydraBase enables failover at the granularity of a single region and reduces failover time by eliminating log splitting.
  82. Saving capacity with HDFS RAIDThis post details the implementation and lessons learned from deploying HDFS RAID (Erasure Codes) to reduce HDFS replication factors, leading to tens of petabytes of capacity savings. It introduces XOR and Reed-Solomon erasure codes, explains the 'small file problem' and the solution of 'directory RAID', and discusses challenges like data corruption, parallelizing RAID for large directories, and handling directory changes. Future work includes native HDFS RAID support.
  83. Scaling the Facebook data warehouse to 300 PBThis post details the evolution of Facebook's data warehouse storage format from RCFile to ORCFile to handle 300 PB of data. Key contributions include: the introduction of ORCFile with 256MB stripes, adaptive column encodings (dictionary encoding for strings and large integers based on heuristics like distinct value count and character set), optimizations to the ORCFile writer for improved write performance (e.g., using hash maps for dictionaries, reducing Zlib compression level), and enhancements to the ORCFile reader for lazy column decompression and decoding to speed up selective queries. These improvements resulted in an overall compression ratio increase from 5x to 8x and a 3x average improvement in write performance compared to open-source ORCFile.
  84. WebScaleSQL: A collaboration to build upon the MySQL upstreamThis post announces WebScaleSQL, a collaboration among Facebook, Google, LinkedIn, and Twitter to build upon the upstream MySQL branch. Key contributions include an automated framework for running and publishing MySQL's built-in test system (mtr) results, a new suite of stress tests and a prototype automated performance testing system, changes to existing tests and code structure to improve maintainability, performance improvements such as buffer pool flushing enhancements and query optimizations, and new features like `super_read_only` and sub-second client timeouts. Future work includes contributing an asynchronous MySQL client, moving production-tested table, user, and compression statistics into WebScaleSQL, and adding the Logical Read-Ahead mechanism.
  85. Under the Hood: Building and open-sourcing RocksDBThis post introduces RocksDB, an embeddable, persistent key-value store developed and open-sourced by Meta. It addresses the challenge of storing and accessing hundreds of petabytes of data efficiently for real-time, user-facing applications. RocksDB is built on LevelDB and is designed to scale on multi-core servers, efficiently utilize fast storage (like flash), offer a flexible architecture for innovation, and support various workloads (IO-bound, in-memory, write-once). It aims to reduce network latency by enabling applications to manage their own datasets locally and improve performance by minimizing lock contention and optimizing storage operations.
  86. Presto: Interacting with petabytes of data at FacebookThis post introduces Presto, a new distributed SQL query engine designed for interactive speed at petabyte scale. It details Presto's architecture, which differs from Hive/MapReduce by using a custom in-memory, pipelined execution engine to reduce latency. It highlights Presto's extensibility through connectors for various data sources (HDFS, HBase, Scribe, custom systems) and its implementation in Java with dynamic bytecode compilation. The post also covers Presto's current status, including its production deployment at Facebook, scale to 1,000 nodes, daily processing of one petabyte, and performance improvements over Hive/MapReduce. Future roadmap items and the open-sourcing of Presto are also announced.
  87. Under the Hood: Building posts searchThis post details the engineering challenges and solutions for building a posts search index capable of handling trillions of posts. Key contributions include: collecting data from a varied MySQL schema, indexing data in HBase using Hadoop MapReduce, updating the index via a pub/sub system (Wormhole) that bypasses caches for initial collection but hits them for updates, serving a large index from a combination of RAM and SSDs, and implementing a sophisticated ranking system with query rewriting and dynamic result scoring based on over a hundred features. The project originated from a hackathon and involved significant infrastructure development.
  88. Under the hood: MySQL Pool Scanner (MPS)Introduced MySQL Pool Scanner (MPS), a state machine system written in Python to automate routine MySQL DBA tasks for Meta's large-scale database clusters. MPS manages instance states (production, spare, drained, etc.) and uses 'problems' (tags) to drive automated actions like instance replacement and primary promotion, enabling bulk maintenance operations with minimal human intervention. It relies on a central repository for state management and integrates with Xtrabackup for data copying and Windex for reimaging.
  89. Flashcache at Facebook: From 2010 to 2013 and beyondThis post details the evolution and optimization of Flashcache, a software-based caching solution implemented as a Linux kernel device mapper target. It describes its initial adoption in 2010 to address storage cost and performance trade-offs, and subsequent performance analysis and improvements made between 2010 and 2013. Key contributions include enhancing read-write distribution through changes in associativity and hashing, optimizing cache eviction by adopting LRU with mid-point insertion, and improving write efficiency with a unified dirty data eviction method. These optimizations led to significant reductions in disk I/O operations, improving the performance and efficiency of Facebook's storage infrastructure.
  90. Scaling Apache Giraph to a trillion edgesThis post details the significant enhancements made to Apache Giraph to scale it to handle trillions of edges. Key contributions include: flexible vertex/edge based input allowing loading from separate sources and multiple Hive tables; HiveIO for up to 4x faster reading/writing to Hive tables; multithreading for graph loading, computation, and result storage; memory optimizations via serialization of vertices/edges into byte arrays and using Java primitives with FastUtil; and sharded aggregators implemented with Netty to bypass Zookeeper limitations and handle large-scale global computations efficiently.
  91. TAO: The power of the graphIntroduced TAO (The Associations and Objects), a graph-based data storage system designed to handle Facebook's demanding social graph workload. TAO provides a unified API for managing objects (nodes) and associations (edges), abstracting away the complexities of distributed caching (Memcache) and persistent storage (MySQL). It addresses limitations of client-side graph APIs by implementing a distributed service for efficient cache management, consistency, and reduced network overhead. TAO partitions data into shards, uses a two-tier caching system (leaders and followers), and optimizes for creation-time locality.
  92. Wormhole pub/sub system: Moving data through space and timeThis post introduces Wormhole, a publish-subscribe system that enables real-time streaming of transactional data from user databases to data warehouses and bulk processing systems like Hadoop, Hbase, and Hive. This replaces the previous reliance on periodic database dumps, significantly reducing latency and improving data freshness for analytics. Wormhole also contributes to cache consistency and provides efficient, reliable, and ordered delivery of updates across data centers, processing over a trillion messages daily.
  93. Scaling memcache at FacebookThis post details the critical role of memcached in scaling Facebook's infrastructure to support over a billion users. It highlights how memcached was adopted early on to address increasing database query loads and page load times, enabling a fast user experience. The post emphasizes the unique challenges of scaling a social network, including the interconnected social graph, large fan-outs, and the need to adapt to rapid product development. It showcases the significant effort invested in scaling the cache infrastructure to handle extremely high request rates and ever-changing products, with memcache being a major component.
  94. LinkBench: A database benchmark for the social graphIntroduced LinkBench, a new database benchmark designed to accurately simulate Facebook's social graph workload. LinkBench replicates the data model, graph structure, and request mix of MySQL social graph data, focusing on transactional workloads for serving real-time queries and updates. The benchmark was developed by analyzing production workload traces to understand out-degree distributions, read/write access patterns (power-law distributions), and the impact of high-degree nodes. It includes a Java driver for generating graphs and operations, with phases for loading and requesting data, and provides adapters for different database systems. Benchmark results for MySQL 5.1.53 are presented, showing throughput and latency under load, along with analysis of server resource utilization.
  95. Under the Hood: Building out the infrastructure for Graph SearchThis post details the development of Unicorn, a unified, scalable inverted-index search engine designed to support Graph Search. It describes the migration of legacy search systems (PPS, Typeahead) to Unicorn, the challenges of indexing the Facebook graph (nodes and edges with metadata), and the design of Unicorn's index structure (posting lists, static rank) and query language. It highlights Unicorn's ability to handle multi-hop queries and the real-time update pipeline for indexing over 2.5 billion new pieces of content daily.
  96. McDipper: A key-value cache for Flash storageThis post introduces McDipper, a highly performant flash-based cache server that is Memcache protocol compatible. It addresses the challenge of efficiently caching large datasets with moderate to low request rates by leveraging flash storage. McDipper is presented as a drop-in replacement for Memcached, offering improved capacity and performance. The post details the technical challenges in achieving feature parity with Memcached, including handling race conditions between sets and invalidations, and introduces solutions like delete hold-offs and read-modify-write operations. A significant application of McDipper is highlighted in Facebook's photo infrastructure, where it serves over 150 Gb/s from forward caches in the CDN, drastically reducing server counts and improving latency. The post also emphasizes the importance of configurability and collaboration in the successful deployment of such infrastructure.
  97. Under the Hood: Automated backupsThis post details the implementation of a three-stage automated backup system for Facebook's large-scale MySQL installations. Stage 1 involves rack backups (RBUs) collecting binary logs and using mysqldump for nightly logical backups. Stage 2 utilizes Hadoop DFS for stable, replicated storage of these backups. Stage 3 involves weekly copying to discrete storage in a separate region. The post also describes a novel exponential scoring system for backup failure severity monitoring and a continuous restore testing process with a self-service restore coordinator (ORC).
  98. Under the Hood: Scheduling MapReduce jobs more efficiently with CoronaIntroduced Corona, a new scheduling framework for MapReduce jobs that addresses limitations in Hadoop's original scheduling framework. Corona separates cluster resource management from job coordination by introducing a cluster manager and dedicated job trackers per job. It uses a push-based scheduling model to minimize latency and implements fair-share scheduling for better fairness guarantees and multi-tenant support. The deployment involved a staged rollout across the production cluster, migrating workloads team by team. Key benefits include improved slot refill times (17% improvement in production, 600ms in simulation), higher cluster utilization (over 95% vs 70%), reduced unfairness (14.3% to 3.6%), and halved job latency for a test job. Future work includes resource-based scheduling and integrating other applications like Peregrine.
  99. Monitoring cache with ClaspinIntroduced Claspin, a monitoring tool that uses heat maps to visualize the health status of Facebook's cache systems (Memcache and TAO). Claspin encodes tribal knowledge about metric 'badness' into a visual format, allowing engineers to quickly assess cache health at a glance. It processes server-side stats and renders heatmaps in the browser using SVG, with real-time updates. Enhancements include support for additional backends, custom palettes, and revision-controlled configuration files.
  100. Under the Hood: Hadoop Distributed Filesystem reliability with Namenode and AvatarnodeThis post details the development and open-sourcing of Avatarnode, a highly available Namenode solution for Hadoop Distributed Filesystem (HDFS). It addresses the single point of failure (SPOF) issue with the traditional Namenode, which was responsible for 41% of HDFS-related incidents in Facebook's Data Warehouse. Avatarnode uses ZooKeeper to manage failover between a Primary and Standby Namenode, ensuring continuous operation and reducing planned downtime. The post also highlights the significant load on the Namenode in large-scale HDFS deployments and the impact of Namenode unavailability on cluster functionality.
  101. MySQL and Database Engineering: Mark CallaghanThis post details Mark Callaghan's work on optimizing MySQL for Facebook's scale, focusing on challenges with multi-core servers and fast storage. Key contributions include adapting innoDB compression for write-heavy workloads and faster storage, improving performance bottlenecks in the database server (e.g., evictions per second), and optimizing checksum algorithms (e.g., using CRC32) with on-the-fly upgrades.
  102. Meet a Facebook Engineer: Haiping ZhaoThis post details the engineer's work on a distributed SQL database designed to scale beyond single-machine limitations and handle diverse data access patterns, aiming to address future challenges in managing large datasets.
  103. Moving an Elephant: Large Scale Hadoop Data Migration at FacebookThis post details a large-scale data migration of dozens of petabytes of Hadoop data between data centers. It describes the decision to use a replication system over a physical move to minimize downtime. The replication process involved a bulk copy using DistCp with custom modifications for Facebook's dataset (handling multiple mappers for large files and directories with many small files) followed by a custom Hive plug-in to record changes in an audit log, which a new replication system then polled to copy modified files and metadata. The switchover involved stopping the JobTracker, allowing replication to catch up, and then updating DNS entries. Challenges included developing a multi-threaded replication system capable of handling millions of objects and managing the numerous systems interacting with the MapReduce cluster during the switchover. The replication system also demonstrated potential for disaster recovery.
  104. Working with Students to Improve Indexing in Apache HiveThis post details the collaboration with Harvey Mudd Computer Science Clinic students to enhance Apache Hive's indexing capabilities. The students implemented automatic index usage and a new bitmap indexing type, which was tested on smaller scales with encouraging results. The work aims to accelerate complex queries, such as those involving multiple attributes and time ranges, and demonstrated significant space savings and query speedups with bitmap index compression.
  105. Building Realtime InsightsThis post details the engineering of a real-time insights system for Facebook Insights for Websites. It describes the challenges of processing 20 billion events per day with sub-30-second latency, including the failure of MySQL counters and in-memory counters due to write rate and reliability issues, respectively. The chosen architecture utilizes HBase for its high write rate and reliability, scribe for log aggregation built on HDFS, and a client for batching and writing data streams from scribe to HBase. Key architectural considerations include table schema, key composition, batching nuances, and sharding.
  106. Hybrid Incremental MySQL BackupsThis post details enhancements to Facebook's database backup system, migrating from slow, logical mysqldump to a custom physical backup model using XtraBackup. Key improvements include: supporting fast table-level restore, enhancing full and incremental backups (including remote streaming), and developing a 'hybrid incremental' approach that uses a page tracker to avoid reading unmodified pages while optimizing IO calls. These changes significantly reduce backup times and space requirements.
  107. Join Optimization in Apache HiveThis post details optimizations to join operations in Apache Hive, a data warehouse infrastructure built on Hadoop. It introduces two key improvements: HIVE-1641, which uses the Hadoop Distributed Cache to efficiently distribute small join tables to mappers, and HIVE-1642, which automatically converts common joins to map joins based on input table sizes during execution. Performance benchmarks show significant speedups (12-26x for optimized map join, 57-163% for auto-converted joins). Future work includes dynamic replication factor determination for compressed hashtable files, smarter table size estimation using sampling, and memory-efficient hashtable implementations.
  108. Visualizing FriendshipsThis post details the process of visualizing friendship locality using data from Apache Hive. It describes sampling 10 million friend pairs, combining this data with user city information to sum friendships between city pairs, and then merging with city longitude and latitude. The post outlines the iterative process of visualization in R, including initial attempts with simple line plotting, the challenges of rendering dense data, and the eventual solution of using weighted lines based on Euclidean distance and friendship count, with a color ramp to represent weight. It also mentions replacing lines with great circle arcs for more accurate geographical representation.
  109. Looking at the code behind our three uses of Apache HadoopThis post details Meta's use of Apache Hadoop in three key areas: data warehousing for web analytics, real-time application serving (with HBase and Scribe), and MySQL database backups. It announces the open-sourcing of their production-tuned Hadoop 0.20 branches. Key technical contributions and features highlighted include: enhancements to the NameNode's locking model for scaling to 30 petabytes, integration of Scribe for near real-time data streaming into Hadoop, and the implementation of reliable HDFS file-append operations for zero data loss with HBase. Specific innovations discussed are the AvatarNode for Hadoop High Availability (contributed as HDFS-976), HDFS-RAID with XOR-parity for storage reduction (contributed to Hadoop MapReduce), and the file-append feature for HBase (contributed to the 0.20-append branch). Future work includes cross-data center Hadoop functionality (HDFS-1432), resource-aware scheduling, slotless MapReduce, HDFS snapshots, and downtime-free JobTracker updates.
  110. The Full Stack, Part IThis post contributes to the understanding of data processing and performance by introducing a comprehensive framework for analyzing system performance based on data characteristics and hardware I/O. It details the hierarchy of computer memory and storage, explaining their latency differences and impact on performance. The post also provides practical advice on performance analysis through estimation and understanding fundamental bottlenecks.
  111. The Life of a Typeahead QueryThis post details the engineering behind Facebook's interactive typeahead search feature, showcasing how the data warehousing and analytics platform evolved to support real-time, low-latency query processing. It describes the architecture involving front-end bootstrapping, AJAX requests, an aggregator service, and specialized leaf services (global and graph) that query indices. The post highlights the integration of caching (memcached) and various data stores (MySQL), the challenges of balancing performance, recall, and relevance, and the iterative process of testing and refinement to achieve a production-ready system for millions of users.
  112. Intro to Facebook SearchThis post details the foundational aspects and complexities of Facebook's search product, highlighting its critical role in user engagement and social graph navigation. It introduces the concept of user-centric search, emphasizing personal and social context as key relevance indicators. The post outlines significant technical challenges, including ranking on the critical path, the lack of traditional query caching due to user-specific searches, the need for a large 'hot' index, and the complexities of live data updates. It also touches upon the expansion of search beyond people to include content, bands, and applications, and the infrastructure required for indexing public and friend content.
  113. Hive – A Petabyte Scale Data Warehouse using HadoopThis post introduces Hive, a petabyte-scale data warehousing framework built on Hadoop, developed at Facebook. It addresses the limitations of traditional RDBMS for rapidly growing data by providing a SQL-like interface over Hadoop's MapReduce. Key architectural components include a Metastore for metadata, a query compiler and execution engine that translates SQL to MapReduce jobs, and SerDe/ObjectInspectors for data format handling. Hive supports tables, partitions, bucketing, and user-defined types and functions. It also extends SQL with keywords like MAP, REDUCE, and TRANSFORM for custom computations, enabling users to plug in scripts for tasks not easily expressible in SQL, such as machine learning model generation.
  114. Facebook’s Scribe technology now open sourceThis post introduces Scribe, a unified system developed to address the challenge of collecting billions of messages daily from Facebook's servers for various use cases like access logs, performance statistics, and News Feed actions. Scribe was designed for scalability, flexible network topology, and a balance of reliability suitable for logging. Its simple data model (category and message) and implementation using Thrift are highlighted as key design decisions that facilitated its widespread adoption and ease of development, contributing to the broader data collection and processing infrastructure.
  115. HadoopThis post details Facebook's early adoption and integration of the open-source Hadoop framework for large-scale data storage and processing. It highlights the use of Hadoop's distributed file system and map-reduce paradigm to enable previously impossible computational projects, leading to improved user experience and publicly released features. The post also introduces Hive, an in-house data warehousing layer over Hadoop that adds features like partitioning, sampling, and indexing, with plans for an open-source release.