Scalability and Large-Scale Systems Engineering

Sep 2026Open-Sourcing Rebalancer: A Generic, High-Performance Library for Solving Assignment ProblemsOpen-sourced Rebalancer, a generic, high-performance library for solving assignment problems. Rebalancer separates concerns of problem specification, storage, solving, and debugging. It provides a language for describing problems using objects, bins, constraints, and objectives, transforming them into expression graphs. It supports both optimal solvers (MIP) and local search solvers, with local search being preferred for large-scale problems at Meta. Rebalancer has been used internally for over nine years to solve millions of assignment problems daily across various infrastructure optimization tasks. A debugging UI tool, Rebalancer Explorer, is also released.

Sep 2026ZGateway: Learnings from Putting a Proxy in Front of ZippyDBThis post introduces ZGateway, a new proxy layer for ZippyDB, Meta's key-value store. ZGateway unifies client traffic, providing admission control, load balancing, and cross-region resilience. It addresses the scalability and reliability issues of ZippyDB's direct-access model by collapsing the many-to-many connection mesh into bounded hops, significantly reducing per-host connection counts and changing the scaling behavior to be independent of the client population. ZGateway also implements request batching and coalescing across clients, amortizing RPC overhead, reducing QPS and CPU, and preventing hot key stampedes. This enables the retirement of client-side batching libraries. The post details ZGateway's architecture, its request path, and its evolution, highlighting its role in traffic routing, tenant isolation, and admission control.

Aug 2026MTIA 300: Meta’s First Training Chip with Built-in NICs and Communication-Offloading EnginesThis post introduces MTIA 300, Meta's first custom AI training and inference accelerator designed for recommendation and ranking models. Key technical contributions include: 1. Integrated networking: Two network chiplets with six custom 800 Gbps RDMA NICs each provide 1.2 TB/s of total I/O bandwidth, eliminating the PCIe bus bottleneck. 2. Communication offloading: 16 dedicated message engines (MEs) with RISC-V cores, NIC interfaces, and near-memory compute (NMC) blocks handle communication independently, delivering over 2.8 TB/s of reduction throughput and achieving near-perfect isolation from the compute grid. 3. Compiled-communication model: The HCCL library compiles collectives into autonomous execution subgraphs dispatched to MEs, removing host involvement after initial instruction copying. 4. Performance improvements: MTIA 300 achieves up to 940 GB/s of communication bandwidth within a rack and demonstrates a 3.9x faster total communication time compared to equivalent GPU clusters for a large production model. The design principles of integrated networking and offloaded collective execution are highlighted as foundational for future AI workloads.

Aug 2026From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta’s Ads RankingIntroduced a multi-stage sequence model architecture for ads ranking that decouples offline user modeling from online ranking tasks. This architecture leverages dense tokenization and target-aware attention to efficiently learn feature interactions directly from data. The model demonstrates LLM-style scaling laws, showing predictable performance improvements with compute. This has led to a cumulative lift of 6% in conversions on Instagram, 3% in conversions on Facebook, and 3.5% in ad clicks on Facebook. The architecture is integrated into Meta's Generative Ads Recommendation Model (GEM).

Aug 2026GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation ModelThis post details how Meta doubled the end-to-end (E2E) training efficiency of its LLM-scale ads foundation model (GEM) to 20-25% Model FLOPs Utilization (MFU) while scaling training FLOPs 4x in 12 months. This was achieved through co-designing kernels, precision, parallelism, networking, and memory. Key innovations include: 1. Compute efficiency: Customized recommendation kernel library (Jagged Flash Attention, Generalized Dot-Product Attention, BlockAttention) and mixed ultra-low precision training (MXFP8 attention and MLP) optimized for recommendation workloads. 2. Scaling efficiency: Topology-aware 5D parallelism with SM-free collectives (2D FSDP + Expert Parallelism for dense parameters, Fully Sharded 2D Model Parallelism for sparse parameters) co-designed with Meta's multi-tiered network hierarchy. The post elaborates on challenges like jagged inputs, diverse interaction patterns, memory-bound operations, and numerical sensitivity in recommendation workloads, and how custom kernels like JFA address padding waste and GDPA unifies asymmetric attention modules.

Jul 2026Exploring Hierarchical Interest Representation For Meta Ads Deep Funnel OptimizationIntroduced Hierarchical Interest Representation, an upstream representation layer for Meta Ads deep funnel optimization. This system uses transformer-based graph learning with bias-aware attention and self-supervised cross-view distillation to learn multi-hierarchical interest representations across a large graph. It enriches sparse interactions by blending real-world knowledge (multimodal advertiser and product content processed through LLMs) with engagement signals, enabling generalization to rare and unseen entities. The system outputs universal embeddings for ads entities and Bag-of-Meaning interest tokens, trained end-to-end on billions of interactions, to power new personalization, retrieval, supervision, and specialized ranking architectures across the ads stack.

Jul 2026Modernizing the Meta Ads Service With an Open-Source Kernel SchedulerThis post details the modernization of the Meta Ads service by implementing a custom scheduling policy using sched_ext, a BPF-based extensible scheduling framework. This initiative was driven by a Linux kernel upgrade that risked latency regressions. The custom policy soft-partitions CPUs into pools for latency-critical and less sensitive work, improving L3 cache locality. The results include a 28% reduction in ads retrieval stage tail latency, 3.28 MW power savings, and a 1.1% increase in ads ranked. Subsequent user-space policy updates further reduced p99 latency by 60% and timeout errors by 18%. The post highlights sched_ext as a strategic asset for parallel and decoupled scheduler optimization, independent deployment, and reduced overheads, and notes its upstreaming into Linux.

Jul 2026Meta’s AI Storage Blueprint at ScaleThis post details the evolution of Meta's BLOB-storage architecture to address the specific demands of AI workloads. Key contributions include: a unified metadata schema backed by ZippyDB for O(1) lookups, elimination of the dataplane proxy in favor of a fat client SDK for direct streaming from Tectonic, and regional deployments colocated with GPUs. It also introduces distributed data caching using spare GPU host memory (leveraging Owl subsystem components) and a read-plan metadata cache (similar to memcache) to absorb spikes and hot spots. Protocol optimizations like hedged reads for laggards and dynamic client-side concurrency control for egress spikes are also discussed.

May 2026SilverTorch: Index as Model — A New Retrieval Paradigm for Recommendation SystemsThis post introduces SilverTorch, a novel retrieval paradigm for recommendation systems that unifies all retrieval components (item index, eligibility filter, scoring layer, user tower) into a single neural network, termed 'Index as Model'. This moves away from a microservice mesh architecture to a pure PyTorch module-based system. Key technical contributions include: redesigning retrieval primitives for GPU execution and model graph integration, implementing a Bloom index filter within the model for efficient, parallelized eligibility checks, and developing a fused Int8 ANN search kernel that reduces memory usage and enables returning a larger pool of candidates. These changes result in significantly higher throughput (23.7x), improved TCO efficiency (20.9x), and better recommendation accuracy by enabling increased modeling complexity within sub-100ms latency budgets.

Apr 2026Modernizing the Facebook Groups Search to Unlock the Power of Community KnowledgeThis post details the modernization of Facebook Groups Search by implementing a hybrid retrieval architecture that combines lexical search (Unicorn inverted index) with semantic search (Faiss ANN search using dense vector representations from a 12-layer, 200-million-parameter model). It introduces an L2 ranking stage using a Multi-Task Multi-Label (MTML) architecture to jointly optimize for clicks, shares, and comments. Additionally, it describes the integration of automated offline evaluation using Llama 3 to grade search results, addressing the challenges of validating semantic search quality at scale. Future work includes applying LLMs in the ranking stage and developing adaptive retrieval strategies.

Apr 2026Capacity Efficiency at Meta: How Unified AI Agents Optimize Performance at HyperscaleIntroduced a unified AI agent platform for Meta's Capacity Efficiency Program. This platform leverages encoded domain expertise and standardized tool interfaces to automate the finding and fixing of performance issues, both proactively (offense) and reactively (defense). The system has recovered hundreds of megawatts of power by compressing manual investigation time from hours to minutes and automating the creation of pull requests for resolutions. The architecture uses a two-layer approach: MCP Tools for standardized interfaces and Skills for encoding domain expertise, enabling both offense and defense use cases with shared infrastructure.

Apr 2026How Meta Used AI to Map Tribal Knowledge in Large-Scale Data PipelinesIntroduced a pre-compute engine with specialized AI agents to systematically read codebase files and produce concise context files, encoding tribal knowledge. This resulted in structured navigation guides for 100% of code modules, documented over 50 non-obvious patterns, and reduced AI agent tool calls by 40%. The system includes an orchestration layer for natural language routing and self-refreshing mechanisms to maintain context freshness.

Apr 2026KernelEvolve: How Meta’s Ranking Engineer Agent Optimizes AI InfrastructureIntroduced KernelEvolve, an agentic kernel authoring system that autonomously generates and optimizes production-grade kernels for heterogeneous AI hardware (NVIDIA GPUs, AMD GPUs, MTIA chips, CPUs). KernelEvolve treats kernel optimization as a search problem, using a job harness to evaluate candidate kernels and LLMs to reason over platform-specific documentation. This approach compresses weeks of expert engineering time into hours, achieving over 60% inference throughput improvement for the Andromeda Ads model on NVIDIA GPUs and over 25% training throughput improvement for an ads model on MTIA chips. It optimizes across high-level DSLs (Triton, Cute DSL, FlyDSL) and low-level languages (CUDA, HIP, MTIA C++).

Mar 2026Meta Adaptive Ranking Model: Bending the Inference Scaling Curve to Serve LLM-Scale Models for AdsIntroduced the Meta Adaptive Ranking Model to address the inference trilemma for LLM-scale ads recommendation systems. Key innovations include request-oriented computation sharing and sequence scaling to transform scaling costs from linear to sub-linear, Wukong Turbo for structural throughput maximization by stabilizing deep models and reducing network overhead, and holistic latency optimization by offloading feature preprocessing to GPUs. Achieved selective FP8 quantization for high-throughput inference with minimal quality degradation and hardware-aware graph/kernel specialization for optimized execution. Enabled O(1T) parameter scale through memory optimizations for sparse, categorical features.

Mar 2026Friend Bubbles: Enhancing Social Discovery on Facebook ReelsThis post details the technical architecture of Friend Bubbles on Facebook Reels, a feature that enhances social discovery by surfacing content interacted with by friends. It describes the use of two machine learning models for viewer-friend closeness (survey-based and on-platform activity signals) and a friend-content aware ranking system that integrates social-graph signals with video relevance. The ranking system uses multi-task, multi-label models with friend-bubble interaction signals and a new objective to maximize overall video engagement, balancing social interaction and content quality. Client-side optimizations for performance at Reels scale are also discussed, including reusing prefetch windows for metadata retrieval, conditional animation disabling, and conservative thresholds for bubble eligibility to maintain smooth scrolling and low CPU overhead. The impact of the feature is measured by increased video watch time, positive sentiment ratings, and deeper video consumption.

Mar 2026FFmpeg at Meta: Media Processing at ScaleThis post details Meta's migration from an internal FFmpeg fork to the upstream version, driven by the need for advanced features like threaded multi-lane encoding and real-time quality metric computation. Key contributions include upstreaming patches for more efficient threading (influencing FFmpeg 6.0 and 8.0) and enabling 'in-loop' decoding for real-time quality metrics (starting with FFmpeg 7.0). The post also highlights the integration of Meta's custom ASIC, MSVP, through standardized hardware acceleration APIs, demonstrating a commitment to both open-source development and internal infrastructure optimization.

Feb 2026Building Prometheus: How Backend Aggregation Enables Gigawatt-Scale AI ClustersThis post introduces Backend Aggregation (BAG) as a key technology enabling Meta's gigawatt-scale AI clusters like Prometheus. BAG is a centralized Ethernet-based super spine network layer that interconnects multiple spine layer fabrics across data centers and regions, supporting petabit-range capacities. It connects regional networks to Meta's backbone, enabling mega AI clusters. The post details how BAG layers are distributed regionally and interconnected using planar or spread topologies. It explains how BAG connects to L2 fabrics built with Disaggregated Schedule Fabric (DSF) and Non-Scheduled Fabric (NSF), discussing oversubscription ratios. Technical details include the use of modular chassis with Jericho3 (J3) ASIC line cards, eBGP with link bandwidth attributes for routing, UCMP for load balancing, and MACsec for security. Resilience is addressed through failure domain analysis and strategies to mitigate blackholing. The post also highlights the use of deep buffer switches for BAG to handle long cable distances and support PFC.

Nov 2025Zoomer: Powering AI Performance at Meta’s Scale Through Intelligent Debugging and OptimizationIntroduced Zoomer, a comprehensive, automated debugging and optimization platform for AI training and inference workloads at Meta. Zoomer provides deep performance insights, enabling energy savings, workflow acceleration, and efficiency gains across Meta's AI infrastructure. It features an Infrastructure and Platform Layer for enterprise-grade scalability, an Analytics and Insights Engine for specialized analyzers (GPU, CPU, communication, memory), and a Visualization and User Interface Layer for intuitive data presentation. Zoomer supports various profiling triggers, comprehensive data capture (GPU metrics, execution traces, host-level data, application annotations, inference-specific data, communication analysis), and distributed analysis pipelines for straggler detection, bottleneck analysis, critical path analysis, anti-pattern detection, parallelism analysis, memory analysis, and load imbalance analysis. Specialized features include single-click QPS optimization for inference, request-level deep dives, real-time memory profiling, and LLM Zoomer for large-scale GenAI workloads.

Nov 2025StyleX: A Styling Library for CSS at ScaleIntroduced StyleX, a new styling system for Meta's large-scale applications. StyleX acts as a compiler that extracts styles at build time and generates static, collision-free atomic CSS, combining CSS-in-JS ergonomics with static CSS performance. It addresses historical challenges with CSS at Meta, such as collisions, dependency management, and large bundle sizes. Key technical contributions include the Babel plugin compiler, atomic CSS generation, stylex.create() for defining static styles, stylex.props() for merging and deduplicating styles, and APIs for shareable values (defineVars, defineConsts) and theming. The system enforces constraints to encourage predictability and enable composition, while preserving CSS features like media queries and pseudoclasses. It also supports dynamic styles via CSS variables and provides APIs for animations and view transitions.

Nov 2025Video Invisible Watermarking at ScaleThis post details the engineering challenges and solutions for scaling invisible watermarking at Meta. It describes the transition from GPU-based inference to a CPU-optimized solution, highlighting the identification and mitigation of bottlenecks related to data transfer, inference latency, and model loading. The post also discusses optimization considerations and trade-offs between latency, watermark detection accuracy, visual quality, and compression efficiency (BD-Rate), and introduces a novel frame-selection method to manage BD-Rate impact and a custom post-processing technique for visual quality evaluation.

Oct 2025Disaggregated Scheduled Fabric: Scaling Meta’s AI JourneyThis post introduces Disaggregated Scheduled Fabric (DSF), Meta's next-generation network fabric technology designed for AI training networks. It details the challenges with traditional IP fabrics for AI workloads (elephant flows, low entropy, suboptimal utilization) and how DSF addresses them through a disaggregated architecture (interface nodes and fabric nodes), packet spraying, and a credit-based congestion control algorithm with Virtual Output Queuing (VOQ). The post also describes the scaling of DSF from single AI zones to dual-stage fabrics (L2 zones) and regional interconnectivity (L3 super-spine), highlighting the use of FBOSS and OCP-SAI. A key feature, Input Balanced Mode, is explained as a mechanism to prevent congestion during link failures by dynamically adjusting traffic based on reduced reachability.

Oct 2025Branching in a Sapling MonorepoIntroduced and open-sourced directory branching in Sapling, a source control system for Meta's monorepo. This new branching model addresses scalability and developer experience tradeoffs by allowing directory-level branching within a single repository. Key technical contributions include the `sl subtree copy`, `sl subtree graft`, and `sl subtree merge` commands, which leverage metadata to reconstruct relationships between directory branches and enable three-way merges. The implementation integrates with build systems like Buck2 via config modifiers and addresses code search challenges. The post details use cases such as managing development and production versions, facilitating experimental development, and unblocking Git migrations.

Oct 2025How Meta Is Leveraging AI To Improve the Quality of Scope 3 Emission Estimates for IT HardwareThis post details Meta's use of AI, specifically NLP and LLMs (Llama 3.1), to improve Scope 3 emission estimates for IT hardware. It introduces a methodology for identifying similar components using TF-IDF and Cosine similarity to leverage existing Product Carbon Footprints (PCFs). It also describes using LLMs to extract data from heterogeneous sources for parameterized modeling and to apply generative AI as a categorization algorithm for creating a new, standard taxonomy of IT hardware emissions. The post highlights the benefits of these AI approaches in enhancing data quality, consistency, and traceability for carbon footprint calculations, ultimately aiming to drive sustainable practices in the IT hardware industry.

Oct 2025OCP Summit 2025: The Open Future of Networking Hardware for AIThis post details advancements in Meta's data center networking for AI training clusters, including the evolution of Disaggregated Scheduled Fabric (DSF) to a dual-stage architecture supporting up to 18,432 XPUs, and the introduction of a new Non-Scheduled Fabric (NSF) architecture based on shallow-buffer Ethernet switches for large AI clusters. It also announces new OCP switch platforms (Minipack3N based on NVIDIA Spectrum-4), updates to FBOSS and SAI to support these fabrics, and new optical solutions (2x400G FR4 LITE, 400G DR4, 2x400G DR4). The post highlights Meta's leadership in the new Ethernet for Scale-Up Networking (ESUN) initiative within OCP, aiming to advance Ethernet for AI system interconnects.

Oct 2025Introducing OpenZL: An Open Source Format-Aware Compression FrameworkIntroduced OpenZL, a new open-source data compression framework designed for structured data. OpenZL applies a configurable sequence of transforms to structured data to reveal patterns before compression, achieving performance comparable to specialized compressors while maintaining the maintenance simplicity of a single universal decompressor binary. It includes an offline trainer stage to automatically generate compression configurations based on data shape descriptions (SDDL) or parser functions, and supports runtime adaptation through control points for dynamic behavior. The framework aims to provide domain-specific compression without ecosystem fragmentation.

Oct 2025Accelerating our Android apps with Baseline ProfilesThis post details Meta's adoption and implementation of Android's Baseline Profiles to enhance the performance of its Android applications. It explains the challenges of ART's class loading and method compilation, Meta's specific performance hurdles with large class sets and frequent updates, and how Baseline Profiles, generated from user data and tuned by Meta, overcome these by enabling install-time optimizations. The post highlights improvements in startup performance and user journeys, contributing to a significant performance uplift across Meta's apps.

Sep 2025LLMs Are the Key to Mutation Testing and Better ComplianceIntroduces the Automated Compliance Hardening (ACH) tool, an LLM-based system for software testing that automates compliance adherence. ACH addresses the challenges of scaling mutation testing by generating fewer, more realistic, and highly specific mutants targeted at particular fault classes. It also features an LLM-based Equivalence Detector agent to identify and filter out equivalent mutants, and automatically generates unit tests that kill the mutants. The tool has been trialed on platforms like Facebook, Instagram, and WhatsApp for privacy testing use cases.

Sep 2025Meta’s Infrastructure Evolution and the Advent of AIThis post details Meta's infrastructure evolution from its early LAMP stack to its current massive, globally networked operations, with a significant focus on the impact of AI workloads. It describes the transition from scaling software systems (Memcache, TAO, News Feed ranking, photo service) to scaling physical infrastructure (data centers, edge infrastructure, backbone networks). The post highlights challenges faced in cache consistency and fleet management, leading to the development of systems like Twine, Tectonic, ZippyDB, Shard Manager, Delos, and Service Router. It also discusses masking hardware failures with systems like Kraken, Taiji, and Maelstrom. The emergence of AI, particularly GPUs for personalized recommendations and LLMs, has driven the need for specialized AI clusters with hundreds or thousands of GPUs interconnected by high-bandwidth, low-latency networks, requiring holistic planning across all infrastructure layers. The post emphasizes the shift from cost-efficient but unreliable commodity hardware to high-performance computational systems optimized for AI.

Aug 2025Enabling Kotlin incremental compilation on Buck2This post details the integration of Kotlin's incremental compiler into Meta's Buck2 build system. Key technical contributions include: 1. Integrating with Kotlin's experimental Build Tools API (KEEP #421) by un-shading the API and later migrating the toolchain to use the shaded kotlin-compiler-embeddable. 2. Configuring Buck2's incremental actions to preserve previous build outputs, enabling the incremental compiler to access necessary artifacts. 3. Ensuring the compiler cache is relocatable by explicitly configuring project and build directories for distributed builds. 4. Implementing change tracking for both module files (using Buck's incremental actions and hash digests) and dependencies (via classpath snapshots and ABI comparison). 5. Adapting custom compiler plugins to be incremental and accumulate results across multiple compilation rounds. 6. Verifying the compatibility of Kotlin Symbol Processing (KSP2) annotation processors with the incremental compiler.

Aug 2025Federation Platform and Privacy Waves: How Meta distributes compliance-related tasks at scaleIntroduces the Federation Platform and Privacy Waves program as a scalable system for managing compliance-related tasks, including privacy, security, and accessibility. Details the technical configuration of workstreams within the Federation Platform, including scraping flags, ownership resolution, grouping, actioning (task/diff generation), task content and distribution, and resolution logic. Outlines the end-to-end federation process, from strategy and task configuration to distribution, accountability, and reporting, highlighting the use of AI for support and summarization, and sophisticated matching algorithms for task assignment.

Jul 2025How Meta keeps its AI hardware reliableThis post details Meta's methodologies for detecting and mitigating Silent Data Corruptions (SDCs) in its AI hardware infrastructure. It categorizes hardware faults into static, transient, and silent errors, highlighting the unique challenges posed by SDCs in AI training (NaN propagation, corrupted gradient variance) and inference workloads. The post introduces and explains three novel SDC detection mechanisms: Fleetscanner, Ripple, and Hardware Sentinel, and discusses their impact on ensuring the reliability of large-scale AI systems.

May 2025Journey to 1000 models: Scaling Instagram’s recommendation systemThis post details the infrastructure evolution at Instagram to support over 1000 ML models in production. Key contributions include the development of a model registry to serve as a central source of truth for model importance and business function, the creation of automated model launch tooling to significantly reduce release times, and the definition and operationalization of 'model stability' as a metric to establish SLOs and drive automated monitoring and alerting. The post also elaborates on the structure of model types and criticality tiers adopted from Meta's Global Service Index to standardize operational responses and improve system-wide observability.

May 2025Meta’s Full-stack HHVM optimizations for GenAIThis post details the optimizations made by Meta's Web Foundation team to the monolithic web tier (running Hack and HHVM) to better handle Generative AI (GenAI) inference traffic. Key contributions include: 1. Creating a dedicated WWW tenant for GenAI traffic to allow custom configurations. 2. Increasing the request runtime limit to accommodate longer inference times. 3. Adjusting thread-pool sizing to handle more concurrent requests. 4. Leveraging HHVM's JIT cache and Jump-Start feature for faster code compilation. 5. Implementing request warm-up to pre-cache configuration and service discovery information. 6. Utilizing request shadowing to ensure JIT cache coverage for future code paths. These efforts led to a 30% reduction in GenAI inference latency.

May 2025Accelerating GPU indexes in Faiss with NVIDIA cuVSThis post details the integration of NVIDIA cuVS into Meta's Faiss library, enhancing GPU-accelerated vector search. It quantifies performance improvements, showing up to 4.7x faster IVF index build times and up to 8.1x reduced search latency for IVF PQ. For graph indexing, CAGRA (via cuVS) offers up to 12.3x faster build times and up to 4.7x reduced search latency compared to CPU HNSW. Benchmarks were conducted on NVIDIA H100 GPUs and Intel Xeon Platinum 8480CL systems using Deep1B and OpenAI text embeddings datasets.

Mar 2025A case for QLC SSDs in the data centerThis post introduces the case for adopting QLC SSDs in Meta's data centers as a new storage tier. It details the limitations of HDDs (declining BW/TB) and TLC SSDs (cost) and positions QLC as a solution offering higher density, improved power efficiency, and better cost. Key technical contributions include: hardware considerations for QLC adoption (U.2 form factor for higher capacity, supporting DFMs), and adaptations to storage software to handle QLC's high density and read/write throughput delta (using ublk over io_uring, tuning rate controllers and I/O schedulers). It also discusses the target workloads (read-bandwidth-intensive with infrequent writes) and the expected benefits in server and rack level byte densification, as well as reduced per-TB acquisition and power costs.

Feb 2025How Precision Time Protocol handles leap secondsThis post details Meta's approach to handling leap seconds within Precision Time Protocol (PTP) environments, which require nanosecond-level precision. It contrasts PTP's needs with traditional NTP leap second smearing methods (quadratic and linear) and introduces Meta's 'self-smearing' algorithm within the fbclock library. The post also highlights the trade-offs of using UTC over TAI for timekeeping in these high-precision systems and advocates for a future without leap seconds to simplify infrastructure and enhance timekeeping precision.

Jan 2025Strobelight: A profiling service built on open source technologyIntroduced Strobelight, a profiling orchestrator that integrates multiple profilers (many eBPF-based) to collect detailed performance metrics (CPU usage, memory allocations) from production hosts. Strobelight enables engineers to identify performance bottlenecks, optimize code, and improve resource utilization, leading to significant capacity savings. It supports ad-hoc profilers via bpftrace scripts and provides default continuous profiling with dynamic sampling and sample weighting for aggregation. Key components contributing to efficiency include the LBR profiler for FDO and the event profiler for stack trace analysis. Stack Schemas enhance data by adding tags to call stacks for better visualization and analysis.

Jan 2025ILA Evo: Meta’s journey to reimagine fiber optic in-line amplifier sitesThis post introduces Meta's 'ILA Evo' effort, a significant initiative to reimagine fiber optic in-line amplifier (ILA) sites. It details the historical context of ILA development and outlines the new requirements for ILA Evo, including faster deployment, reduced material usage, improved power efficiency (PUE < 1.5), and the adoption of lightweight building systems (FRP/GRP panels), low-concrete foundations (helical screw piles), manufactured owner-furnished equipment (MOFE) for ISP installation, advanced cooling technologies, and modern backup power alternatives (H2 fuel cells, molten salt batteries). The post also highlights the macro benefits of this disaggregated supply chain for improved commercial and risk management.

Dec 2024Indexing code at scale with GleanIntroduced Glean, an open-source code indexing system for collecting and querying facts about source code. Detailed Glean's design rationale, emphasizing language and use-case generality, and its use of RocksDB for storage and Angle for querying. Explained incremental indexing to achieve O(changes) indexing cost and discussed its application in code navigation, symbol search, and speeding up IDEs.

Dec 2024Translating Java to Kotlin at ScaleThis post details the development and implementation of the Kotlinator, an automated tool for translating millions of lines of Java code to Kotlin for Meta's Android codebase. It describes the six-phase process, including custom preprocessing and postprocessing steps built on internal metaprogramming tools, and the integration of headless J2K. The post also highlights collaboration with JetBrains to improve J2K for K2 compatibility and enable custom IDE extensions.

Dec 2024How we think about Threads’ iOS performanceThis post details how the Threads iOS team approaches and maintains app performance during rapid growth. It introduces key performance metrics like %FIRE (frustrating image-render experience), TTNC (time-to-network content), and cPSR (creation-publish success rate). It also discusses the development of the SLATE debugger for consistent navigation latency measurement and a case study on validating GraphQL adoption for network requests, highlighting the importance of performance analysis in scaling new features and platforms.

Dec 2024Inside Facebook’s video delivery systemThis post details the technical evolution of Facebook's video delivery system, focusing on the two-year effort to unify Watch and Reels. It describes the three core components: Ranking (RecSys), Server (WWW), and Mobile (FB4A/FBiOS). Key technical challenges and solutions are highlighted, including the unification of client and server data models and architectures, the migration of users to a unified Reels surface, and the unification of ranking layers to handle mixed inventory of short-form and long-form video. The post also outlines the five core principles guiding video delivery and the lifecycle of a video feed network request, from mobile client initiation to server processing, emphasizing prefetching, head loads, and tail loads with caching and user signal integration.

Dec 2024Meta Andromeda: Supercharging Advantage+ automation with the next-gen personalized ads retrieval engineIntroduced Andromeda, a next-generation personalized ads retrieval engine that significantly enhances Meta's ad recommendation system. Andromeda leverages custom-designed deep neural networks optimized for NVIDIA Grace Hopper Superchip and MTIA hardware, achieving +6% recall improvement and +8% ads quality improvement. It employs hierarchical indexing to manage exponential ad creative growth from Advantage+ automation and generative AI, and introduces model elasticity for agile resource allocation, boosting model inference efficiency by 10x. The system achieves over 100x improvement in feature extraction latency and throughput compared to previous CPU-based components, and enhances end-to-end model inference QPS by over 3x.

Nov 2024Sequence learning: A paradigm shift for personalized ads recommendationsThis post details a paradigm shift in Meta's ads recommendation engine, moving from DLRMs with human-engineered features to a sequence learning approach. Key contributions include: 1. Introduction of Event-Based Features (EBFs) to standardize heterogeneous inputs into event streams, define sequence lengths, and capture event information. 2. Development of an event model to synthesize event embeddings from attributes and timestamps, translating EBF sequences into event embedding sequences. 3. Implementation of a sequence model utilizing attention mechanisms (with complexity reduced from O(N*N) to O(M*N)) to synthesize event embeddings into ad-keyed representations. 4. Scaling innovations for sequence learning architectures, including custom transformer architectures and hardware codesign for jagged tensors (native PyTorch support, GPU kernel optimization, Jagged Flash Attention). 5. Scaling with longer and richer sequences through multi-precision quantization, value-based sampling, and multimodal enrichment of event attributes.

Oct 2024OCP Summit 2024: The open future of networking hardware for AIIntroduced Disaggregated Scheduled Fabric (DSF) for next-generation AI clusters, a disaggregated and open network fabric. Contributed two new disaggregated network fabrics (Arista 7700R4 series, Minipack3/Cisco 8501) and a new NIC (FBNIC) to OCP. DSF leverages VoQ-based switched systems, OCP-SAI, and FBOSS for proactive congestion avoidance and high-bandwidth AI clusters. FBNIC is a multi-host foundational NIC with datapath isolation and HW offloads, with its driver upstreamed and module contributed to OCP. Enhanced optics to 2x400G FR4 for 400G/800G interconnection.

Oct 2024Meta’s open AI hardware visionThis post details Meta's contributions to open AI hardware, including the Catalina rack design (ORv3, 140kW, liquid-cooled) for NVIDIA GB200 Grace Blackwell Superchip, and the Grand Teton platform's expansion to support AMD Instinct MI300X accelerators for AI inference. It also introduces the Disaggregated Scheduled Fabric (DSF) for next-generation AI clusters, powered by OCP-SAI and FBOSS, supporting RoCE interfaces with partners like NVIDIA, Broadcom, and AMD. New 51T fabric switches and a Meta-designed NIC module (FBNIC) are also shared. The post emphasizes the need for open hardware to scale AI infrastructure, projecting terabyte-per-second injection bandwidth per accelerator and equal normalized bisection bandwidth.

Oct 2024React at Meta Connect 2024This post details how Meta leveraged React and React Native to build and scale applications for Meta Quest, including Instagram and Facebook for Meta Quest, and the Meta Horizon mobile app. It highlights the use of React Native for its animation capabilities, performance, and shared platform for 2D Meta Quest system apps. Code sharing technologies like StyleX and React Strict DOM were used to reuse features from Facebook.com. Performance improvements in the Meta Horizon app were achieved by initiating network queries earlier using Relay. The Meta Horizon Store benefited from React and React Native for cross-platform development and feature parity. The Meta Spatial Editor utilized React Native for Desktop, enabling native integrations and a positive developer experience for C++ engineers. The post emphasizes Meta's ongoing investment in React infrastructure, including the React Compiler and New React Native Architecture, to enable cross-platform code sharing and performance enhancements.

Aug 2024RoCE networks for distributed AI training at scaleThis post details Meta's development of specialized data center networks for distributed AI training, focusing on the implementation and evolution of RoCEv2 for inter-GPU communication. It covers the network topology (AI Zones, aggregator switches), routing challenges and solutions (ECMP, path pinning, Enhanced ECMP with QP scaling), and congestion control strategies (PFC, receiver-driven traffic admission) to support large-scale AI workloads like LLMs.

Aug 2024DCPerf: An open source benchmark suite for hyperscale compute applicationsThis post introduces DCPerf, an open-source benchmark suite developed by Meta to represent diverse hyperscale compute workloads. It details the methodology used to ensure benchmark representativeness by analyzing production workloads for low-level hardware microarchitecture features and application/library usage profiles. The post highlights DCPerf's compatibility with different instruction set architectures (x86, ARM) and its support for multi-tenancy. It also showcases how Meta uses DCPerf internally for product evaluation, capacity planning, identifying performance bugs, and collaborating with hardware industry partners to optimize future server designs, demonstrating its superior coverage and accuracy compared to existing benchmarks like SPEC CPU in representing production application IPC and power/frequency characteristics.

Jul 2024Taming the tail utilization of ads inference at Meta scaleThis post details Meta's efforts to tame the tail utilization of ads inference services. Key technical contributions include: leveraging the 'power of two choices' in randomized load balancing via ServiceRouter with polling for fresh load data, tuning Shard Manager placement load balancing configurations (load bands, thresholds, balancing frequency) to address spiky tail utilization, and incorporating memory bandwidth as a resource during replica placement in Shard Manager to mitigate CPU stalls caused by memory latency. The post also highlights the mismatch in expectations between ServiceRouter's host-level consolidated load counter and Shard Manager/ReplicaEstimator's assumption of equal replica load, leading to imbalance issues.

Jul 2024Meta’s approach to machine learning prediction robustnessThis post details Meta's systematic framework for building prediction robustness in its machine learning systems, specifically for advertising recommendations. It outlines the challenges of ML robustness (stochastic nature, frequent model refreshes, blurred reliability/performance lines, cumulative effects, complex interactions, amplification of small changes, and rapid innovation). The framework includes prevention guardrails, fundamental understanding, and intrinsic robustness applied across models, features, training data, calibration, and interpretability. Key technical contributions include Snapshot Validator for model snapshot quality, new ML techniques for intrinsic model robustness (pruning, generalization, quantization), robust feature monitoring systems, dedicated training-data-quality systems, high-precision alert systems for calibration, and the Hawkeye AI debugging toolkit for ML interpretability. These efforts aim to improve ML ranking performance, boost engineering productivity, and reduce the time to root-cause ML prediction issues.

Jun 2024PVF: A novel metric for understanding AI systems’ vulnerability against SDCs in model parametersIntroduced Parameter Vulnerability Factor (PVF), a novel metric for quantifying AI model vulnerability against silent data corruptions (SDCs) in model parameters. PVF is adaptable to different AI models, tasks, hardware fault models, and can be extended to the training phase. The post details the methodology for computing PVF through fault injection experiments and presents case studies demonstrating its application, such as analyzing the vulnerability of DLRM components. It also references Dr. DNA, a method for detecting and mitigating SDCs during deep learning model inference.

Jun 2024How Meta trains large language models at scaleThis post details Meta's approach to training large language models (LLMs) at scale, highlighting the shift from training many smaller models to fewer, extremely large ones. It outlines the key challenges: hardware reliability, fast recovery on failure, efficient state preservation, and optimal GPU connectivity. The post describes innovations across the infrastructure stack, including PyTorch for training software, sophisticated scheduling, hardware modifications to the Grand Teton platform (increased TDP, HBM3), data center deployment strategies for high compute density, and reliability measures for frequent hardware failures (e.g., GPUs falling off, UCEs, network cable issues). A significant focus is placed on network infrastructure, detailing the development and operation of both RoCE and InfiniBand fabrics (24k clusters each) for GenAI training, including optimizations for communication patterns, topology-aware collectives, and network load balancing. It also touches upon storage solutions for large datasets.

Jun 2024Maintaining large-scale AI capacity at MetaThis post details Meta's strategies for maintaining large-scale AI capacity, focusing on the unique challenges of GPU training clusters. It introduces 'maintenance trains' for cyclic capacity updates, gradual rollouts to manage software and firmware updates across heterogeneous hardware, and the OpsPlanner orchestrator for managing disruptive work and ensuring host consistency. Key technical challenges addressed include capacity guarantees, the impact of 'bad hosts', low interruption rates, rollout safety, and host consistency. The post highlights the trade-offs between maintaining a consistent AI job stack and gradually upgrading lower-level components, emphasizing the need for careful testing, monitoring, and vendor collaboration. It also discusses the optimization of maintenance domain sizes based on interruption costs and reserved capacity.

Jun 2024Unlocking the power of mixed reality devices with MobileConfigThis post details the expansion of Meta's MobileConfig system to support mixed reality (MR) devices, including Meta Quest and Ray-Ban Meta smart glasses. It addresses configuration challenges in the MR ecosystem, such as extended development cycles and the need for shared configuration values across multiple apps. The post explains how MobileConfig was adapted by centralizing configuration requests through a single Android service, developing cross-platform libraries, and optimizing for low-powered devices and microcontrollers with limited connectivity and resources. Key technical adaptations include repurposing libraries for microcontrollers, optimizing memory footprint, and developing customized IPC APIs and AOSP Java/Native APIs for OS-level experimentation. The goal is to enable rapid innovation, experimentation, and remote control for future devices.

Mar 2024Building Meta’s GenAI InfrastructureThis post details Meta's new 24k GPU clusters for GenAI, focusing on hardware, network, storage, design, performance, and software. It highlights the use of Grand Teton and OpenRack for compute, dual network fabrics (RoCE and InfiniBand) for interconnectivity, and a home-grown FUSE API backed by Tectonic storage, augmented by Hammerspace for parallel NFS. Performance optimizations for large clusters are discussed, including network topology awareness and NCCL tuning, achieving over 90% bandwidth utilization. Debugging tools like 'desync debug' and PyTorch evolution for large-scale training are also mentioned.

Feb 2024DotSlash: Simplified executable deploymentIntroduced DotSlash, a tool for simplified executable deployment that replaces large executables with lightweight descriptor files. DotSlash handles transparent fetching, decompression, and verification of platform-specific remote artifacts, reducing repository size and improving clone times. The post details the internal use cases at Meta, including managing first-party and third-party command-line tools within a monorepo and for core developer tools like Buck2 and Sapling. It also describes the DotSlash file format, its execution workflow, caching mechanisms, and the automated generation of DotSlash files via CI pipelines and GitHub Actions.

Jan 2024Improving machine learning iteration speed with faster application build and packagingThis post details improvements to the ML/AI development workflow by addressing slow build times and inefficiencies in packaging and distributing executable files. Key contributions include: 1. Mitigating build non-determinism using Buck2 and Remote Execution to ensure consistent build outputs and enable warm revisions. 2. Optimizing dependency management by reducing dependency counts, removing unused code, and identifying used Python modules. 3. Implementing a Content Addressable Filesystem (CAF) for incremental packaging and distribution of Python executables, leveraging a CAS daemon with P2P distribution via Owl and Btrfs for efficient storage. This resulted in double-digit percentage reductions in overhead.

Dec 2023How Meta built the infrastructure for ThreadsThis post details how Meta's existing infrastructure, specifically ZippyDB and Async, was leveraged to support the rapid launch and growth of the Threads application. It highlights ZippyDB's features for scaling keyspaces, including its capacity management framework, multi-tenancy, and resharding protocol, which enabled Threads to scale by 100x with zero downtime. It also explains Async's role as a serverless function platform in handling computationally expensive operations like user onboarding and follow requests, managing massive job volumes and rate limits automatically to avoid overloading downstream services. The post emphasizes the importance of a decade of infrastructure development and a 'move-fast' engineering culture in enabling such rapid product launches and scaling.

Oct 2023Automating dead code cleanupIntroduced the Systematic Code and Asset Removal Framework (SCARF) with a subsystem for identifying and removing dead code. SCARF combines static and dynamic analysis (leveraging Glean for code dependency graphs and operational logs for API usage) across multiple programming languages (Java, Objective-C, JavaScript, Hack, Python). It operates at a symbol level for granular analysis and automatically generates code change requests via CodemodService. The system has evolved to analyze the complete augmented dependency graph, leading to a significant increase in dead code removal. It also incorporates application-specific semantics and domain-specific logging to fine-tune dependency analysis. SCARF has automatically deleted over 100 million lines of code.

Oct 20235 Things you didn’t know about Buck2This post introduces Buck2, the successor to Meta's open-source build system Buck. Key technical details include: Buck2 is written in Rust for performance and memory control, with rules in Starlark. It supports remote execution to avoid downloading intermediate outputs, saving bandwidth and disk space. Buck2 defines 11 distinct file path types for enhanced type safety. It is language-agnostic, with language-specific logic defined in Starlark. Buck2 introduces BXL (Buck2 Extension Language) for inspecting and interacting with the build graph, enabling custom build actions and IDE project generation.

Oct 2023Automating product deprecationIntroduced the Systematic Code and Asset Removal Framework (SCARF), Meta's internal tool and framework for automating product and feature deprecation. SCARF guides engineers through the process of safely and efficiently removing code and data associated with deprecated products. It combines workflow management tools with automation to reduce engineer load, having removed petabytes of unused data and over 100 million lines of code. SCARF analyzes code and data dependencies, identifies internal and external dependencies, and creates deletion roadmaps to ensure a staggered, multi-step removal process. It leverages static and dynamic analysis metrics to determine which assets are in use and which are safe for automated deletion, facilitating a collaborative approach between engineers and automation.

Sep 2023Arcadia: An end-to-end AI system performance simulatorIntroduced Arcadia, Meta's unified end-to-end AI system performance simulator. Arcadia models compute, memory, and network performance of AI training clusters, enabling data-driven design decisions for AI clusters and infrastructure. It addresses challenges in optimizing large-scale distributed systems, multi-layered systems, operational workflows, AI workload characteristics, and the need for a common source of truth by providing a unified simulation framework with inputs from system plans, network topology, workload distributions, and hardware specs. The core orchestrates simulations of job scheduling, compute, memory, and network behavior, using an AI workload synthesizer. Outputs include performance metrics, resource utilizations, and reliability metrics, with a feedback loop for accurate production estimations. Arcadia offers operational insights, flexibility in simulation detail, and serves as a single source of truth for co-design.

Sep 2023Threads: The inside story of Meta’s newest social appThis post details the rapid development and launch of Meta's Threads app, highlighting the engineering challenges and solutions involved in scaling to over 100 million users in a short period. It emphasizes the reuse of existing Python (Django) backend technology, Swift on iOS, and Jetpack Compose on Android. The post also touches upon the critical role of Meta's underlying infrastructure, internal monitoring tools (ODS, Scuba), and the collaborative efforts of infrastructure, product, and on-call engineers during the launch. Furthermore, it outlines the future direction of Threads towards decentralization and interoperability with protocols like ActivityPub.

Aug 2023Scaling the Instagram Explore recommendations systemThis post details the scaling of the Instagram Explore recommendations system using advanced machine learning models, specifically Two Towers neural networks, and a multi-stage ranking approach. It elaborates on the four stages: retrieval, first-stage ranking, second-stage ranking, and final reranking. Key technical contributions include the application of Two Towers NNs for both retrieval and first-stage ranking due to their cacheability, the use of user interaction history for retrieval, the distillation of knowledge from a heavier second-stage model to a lighter first-stage model, and the development of a value model (VM) to combine engagement probabilities for final scoring. The post also highlights the use of caching and pre-computation to handle heavy models and ensure system availability.

May 2023Building and deploying MySQL Raft at MetaThis post details the implementation and benefits of MySQL Raft, a system that integrates the Raft consensus engine into MySQL to manage replication, promotions, and membership changes. It replaces the previous semisynchronous replication and manual automation-driven control plane operations with a unified, provably safe distributed system. Key technical contributions include the use of Apache Kudu's Raft library (forked as kuduraft) with enhancements like FlexiRaft, proxying, and compression. The MyRaft plugin interfaces Raft with MySQL, and the binary log is used as the replicated log. The write path involves Raft assigning OpIds, compressing transactions, and waiting for consensus before committing to the engine. Crash recovery is also adapted to work seamlessly with Raft.

Feb 2023Improving Meta’s global mapsThis post details the evolution of Meta's global basemap since its launch in late 2020. It highlights significant upgrades in data storage, the introduction of dark-mode and raster-first basemaps for faster rendering across devices, and the integration of open-source data from OpenStreetMap (Daylight) and the upcoming Overture Maps Foundation. The post also discusses improvements in cartographic design, including a toned-down color palette, deprioritization of roads to focus on community spaces, and the introduction of detailed pavement features and iconography. AI-based translations for map labels and the development of 3D geospatial map data for the metaverse are also mentioned as future directions.

Jan 2023Asynchronous computing at Meta: Overview and learningsThis post details the re-architecture of Meta's asynchronous computing platform ('Async tier') to enable five-fold growth and support for multiple external data sources. Key changes include decoupling storage reading into 'tailers' and introducing a generic 'scheduler' for transport and routing. The shift from a pull-based system (Dispatcher pulling from FOQS) to a push-based system (tailers pushing to scheduler) improved cross-regional load balancing and end-to-end latency. The architecture now avoids data duplication in FOQS and treats various data sources as first-class citizens. The post highlights the challenges of the previous monolithic dispatcher and the benefits of granular components with clean contracts.

Nov 2022How Precision Time Protocol is being deployed at MetaThis post details Meta's implementation of Precision Time Protocol (PTP) to achieve nanosecond-level time synchronization across its systems, a significant upgrade from the millisecond-level precision of NTP. It outlines the architectural components (PTP rack, network, client), the case for PTP (improving linearizability and performance in distributed systems), and the technical innovations involved, including GNSS-over-fiber technology, a diversified Time Appliance with FPGA-based Time Cards and disciplined Network Cards, and the development of ptp4u for high-scalability PTP serving.

Nov 2022Sapling: Source control that’s user-friendly and scalableThis post introduces Sapling, a new Git-compatible source control client developed at Meta, emphasizing usability and scalability for massive repositories. It highlights Sapling's features like smartlog for repository visualization, simplified mistake recovery, and robust commit stack management. The post also announces the open-sourcing of the Sapling client and previews future plans for open-sourcing the Sapling server and virtual file system, along with the ReviewStack demonstration tool for stack-oriented code reviews.

Sep 2022Scaling data ingestion for machine learning training at MetaThis post details the development of a new data ingestion infrastructure and last-mile transformation pipelines to support the growing demands of AI model training at Meta. Key technical contributions include: 1. Designing a disaggregated Data PreProcessing tier (DPP) for scalable data ingestion and feature transformation. 2. Implementing feature flattening as a new storage format to improve selective feature reading efficiency by 2-2.3x. 3. Introducing DPP client-side rebatching to support larger batch sizes, leading to 20-40% improvements in reader node throughput. 4. Discussing optimizations for memory bandwidth utilization in data reading workloads, including the FlatMaps project.

Sep 2022Open-sourcing TAOBench: An end-to-end social network benchmarkOpen-sourced TAOBench, a new benchmark that accurately simulates social network workloads by capturing transactional requirements, data colocation preferences, request distributions, and multitenant behavior. TAOBench is derived from profiling requests served by TAO, Meta's online graph data store, and provides workload configurations and a benchmarking framework to model production workloads and generate emergent application behavior. The post also details benchmark use cases at Meta and reports results for five popular distributed database systems.

Sep 2022Network Entitlement: A contract-based network sharing solutionThis post introduces 'Network Entitlement,' a contract-based network sharing solution designed to address challenges in managing Meta's growing network usage and traffic volume. It details the framework's five key properties: isolation for reliability, guarantee and accountability, abstraction of network complexity, observability, and work-conserving behavior. The post also outlines the four components of the solution: contract abstraction, dynamic SLO-based guarantee service, runtime enforcement system, and policy verification using production drills. This represents a significant step in proactively managing and scaling network infrastructure to meet increasing demands and ensure service-level objectives.

Sep 2022Viewing the world as a computer: Global capacity managementIntroduced Global Reservations Service and Regional Fluidity to enable region-agnostic service and capacity management across 14 global data centers. Global Reservations simplifies fault tolerance for latency-tolerant services by using a mixed-integer linear programming solver to assign machines to global reservations, optimizing for placement and disaster-readiness buffers. Regional Fluidity addresses latency-sensitive services by modeling demand sources and rebalancing services to minimize cross-region network requests.

Jul 2022It’s time to leave the leap second in the pastThis post highlights Meta's active role in advocating for the discontinuation of leap seconds, a practice that has historically caused significant disruptions and outages in digital systems. It details the technical challenges associated with leap seconds, including the complexities of 'smearing' them to mitigate negative impacts, and emphasizes Meta's support for an industry-wide effort to move beyond this outdated timekeeping mechanism to ensure greater system stability and reliability.

Jul 2022Owl: Distributing content at Meta scaleIntroduced Owl, a new system for high-fanout distribution of large data objects (executables, code artifacts, AI models, search indexes) within Meta's private cloud. Owl addresses the distribution challenge by balancing fanout, size, and hotness requirements, improving download speeds and cache hit rates by 2-3x over prior systems like BitTorrent. It employs a split design with a decentralized data plane (peer libraries) and a centralized control plane (trackers) to manage distribution decisions, offering high customization through modular policies and trace-driven emulation.

Jun 2022Transparent memory offloading: more memory at a fraction of the cost and powerThis post introduces Transparent Memory Offloading (TMO), Meta's data center solution for managing memory costs and power consumption. TMO offloads colder memory pages to cheaper storage technologies like compressed memory and NVMe SSDs. It utilizes a new Linux kernel mechanism that measures lost work due to resource shortages (CPU, memory, I/O) in real-time via Pressure Stall Information (PSI). A userspace agent, Senpai, uses PSI feedback to dynamically adjust the amount of memory offloaded, without application modification. TMO has been in production since 2021, saving 20-32% of memory per server across millions of machines and has had its OS components upstreamed into the Linux kernel.

Jun 2022Introducing Zelos: A ZooKeeper API leveraging DelosThis post introduces Zelos, a new system built on the Delos platform to replace Apache ZooKeeper. Zelos addresses ZooKeeper's scalability limitations by decoupling application state from its consensus protocol (ZAB). Delos provides a linearizable shared log and state machine replication, abstracting away consensus complexities. Zelos bridges the impedance mismatch between ZooKeeper's session management and Delos's abstractions by implementing speculative execution for session ordering and using lightweight logical snapshots for efficient read processing. This migration aims to provide a more modular, extensible, and performant coordination service.

Jun 2022Cache made consistentThis post introduces a methodology and tooling (Polaris) to effectively measure and monitor cache consistency, specifically addressing the challenges of cache invalidation. It details how to define and observe client-observable invariants to detect inconsistencies without false positives, improving the reliability of large-scale cache systems like TAO and Memcache.

May 2022Delta: A highly available, strongly consistent storage service using chain replicationIntroduced Delta, a new object storage service designed for critical systems like build and distribution artifacts, focusing on high availability and strong consistency. Delta utilizes chain replication, organizing servers in a linear chain (head to tail) where writes are pipelined and acknowledged only after persistence by all servers, and reads are served from the tail to guarantee strong consistency. It trades latency and storage efficiency for simplicity and reliability, supporting horizontal scalability by rebalancing chains across new servers. Delta's architecture includes multiple chains per bucket, spread across failure domains, and employs a voting mechanism (2 hosts) to detect and remove faulty hosts, with automated repair flows for recovered hosts.

Mar 2022Augmenting Flexible Paxos in LogDevice to improve read availabilityThis post details improvements to LogDevice's consensus algorithm, specifically augmenting Flexible Paxos to enhance read availability. The core innovation is enabling reads without a full read quorum by guaranteeing a single value exists for a given slot even during leader failure. This is achieved by leveraging LogDevice's recovery process and a separate membership mechanism based on ZooKeeper. The post also discusses the trade-offs of this approach, particularly the potential for linearized writes to be sacrificed for write availability, and the practical challenges encountered with 'stuck recovery' scenarios, leading to the proposal of a flexible leader election quorum.

Jan 2022FOQS: Making a distributed priority queue disaster-readyThis post details the evolution of FOQS from a regional to a geo-distributed, global architecture to enhance disaster readiness. Key technical contributions include: 1. Migrating from a regional MySQL topology to a global one that follows MySQL primaries on failover, ensuring data availability and reducing cross-region latency. 2. Introducing an intelligent routing service with an enqueue distributor that optimizes data placement based on regional preferences and a dequeue distributor that efficiently discovers ready items across queue nodes. This enables seamless failover and continued service availability during disaster events.

Nov 2021OCP Summit 2021: Open networking hardware lays the groundwork for the metaverseThis post announces two new milestones for Meta's data centers: the sharing of its next-generation network hardware portfolio (Wedge 400/400C TORs and Minipack2/Arista 7388X5 fabric switches) and the migration of its data center network hardware to the Open Compute Project (OCP) Switch Abstraction Interface (SAI). It highlights the continued commitment to openness and disaggregation, building on a decade of data center design and founding OCP. The post emphasizes how these advancements in open networking hardware are laying the groundwork for future technologies like the metaverse and advanced AI applications.

Oct 2021Kangaroo: A new flash cache optimized for tiny objectsIntroduced Kangaroo, a novel flash cache design optimized for tiny objects (~100 bytes or less). Kangaroo combines log-structured (KLog) and set-associative (KSet) cache components to reduce both DRAM overhead and write amplification. It employs a threshold admission policy to amortize writes to KSet over multiple objects and a new eviction policy to minimize cache misses. Evaluated using production social graph caching workloads from Facebook and Twitter, demonstrating a 29% reduction in misses under realistic constraints compared to prior designs.

Sep 2021Superpack: Pushing the limits of compression in Facebook’s mobile appsIntroduced Superpack, a novel compression technique that combines compiler analysis with data compression to reduce the size of mobile applications. Superpack achieves over 20% savings compared to default Android Zip compression by applying advanced LZ parsing and entropy coding techniques informed by compiler analysis of code structure (ASTs, opcodes). It also transforms unstructured data into programs for compression. Superpack is applied to Dex bytecode, ARM machine code, and Hermes bytecode.

Sep 2021CacheLib, Facebook’s open source caching engine for web-scale servicesThis post announces the open-sourcing of CacheLib, a pluggable in-process C++ caching engine designed for building and scaling high-performance services. It details CacheLib's API for creating customizable, scalable, and concurrent caches, and highlights its adoption in over 70 large-scale systems at Facebook. The post also introduces CacheBench, a benchmarking tool for evaluating caching performance on diverse production workloads. It emphasizes the potential for CacheLib and CacheBench to become industry standards, citing collaborations with Twitter, Pinterest, and academic institutions, as well as contributions to the Open Compute Project (OCP) NVMe Cloud SSD Specification.

Aug 2021RAMP-TAO: Layering atomic transactions on Facebook’s online graph storeIntroduced RAMP-TAO, a new protocol that layers atomic transaction capabilities on top of Facebook's TAO online graph store. RAMP-TAO provides stronger transactional guarantees for recently updated data in an eventually consistent system, enabling an intuitive read transaction API for developers. It achieves this with minimal memory overhead (0.42%) and maintains performance comparable to existing TAO reads, with over 99.9% of reads completing in one round-trip to the local cache. The protocol is designed to be cache-friendly, hotspot tolerant, extensible, and incurs overhead only for opt-in applications.

Aug 2021Open-sourcing a more precise time applianceThis post details the development and open-sourcing of the Open Compute Time Appliance and the associated Time Card. The Time Card is a PCIe card integrating a GNSS receiver, a miniaturized atomic clock (MAC), and an FPGA to provide nanosecond-level time interpolation. It addresses the limitations of off-the-shelf time appliances by offering open-source software, user-serviceable hardware, and improved security. The Time Appliance, when combined with an open-sourced time server, significantly enhances timekeeping accuracy in data centers, improving infrastructure management and distributed database performance.

Aug 2021How we built a general purpose key value store for Facebook with ZippyDBThis post details the history and architecture of ZippyDB, Meta's largest strongly consistent, geographically distributed key-value store. ZippyDB reuses RocksDB as its storage engine and integrates with Data Shuttle for replication, Shard Manager for orchestration, and ZooKeeper for configuration. It employs a tiered architecture with regional replication (Paxos/async) and configurable consistency levels. The introduction of micro-shards (μshards) and Akkio mapping allows for transparent resharding and efficient regional data placement. The post also covers ZippyDB's support for transactions and conditional writes.

Jul 2021A linear programming approach for optimizing features in ML modelsThis post introduces a linear programming framework to optimize feature selection for ML models based on infrastructure constraints (memory, CPU, storage). It models the problem as maximizing model gain subject to capacity limitations of various services. The approach allows for quantifying the impact of adding capacity, understanding inter-service dependencies, and automating feature approval decisions. The implementation details using Python with NumPy and CVXPY are provided.

Jul 2021Migrating Facebook to MySQL 8.0This post details the technical challenges and solutions involved in migrating Facebook's core database infrastructure from MySQL 5.6 to MySQL 8.0. It covers the process of porting over 1,700 custom code patches, categorizing them into 'Drop', 'Build/Client', 'Non-MyRocks Server', and 'MyRocks Server' for systematic migration. The document outlines a phased migration strategy for replica sets, emphasizing the standardization on row-based replication (RBR) and the development of new automation infrastructure to manage the transition. It also highlights the validation process, including testing for automation compatibility and application-level regressions, such as issues with text output parsing, collation mismatches, changed error codes, obsoleted .frm files, dynamic privileges, new reserved keywords, and REGEXP incompatibilities.

Jul 2021Fully Sharded Data Parallel: faster AI training with fewer GPUsIntroduced Fully Sharded Data Parallel (FSDP) as a new tool for large-scale AI model training. FSDP shards model parameters, gradients, and optimizer states across data parallel workers, reducing GPU memory requirements and improving computational efficiency through communication and computation overlapping. The post details the technical implementation of FSDP, explaining how it decomposes all-reduce operations into reduce-scatter and all-gather, and provides usage examples for language models, computer vision models, PyTorch Lightning, and the FairScale library directly.

Jul 2021How WhatsApp enables multi-device capabilityThis post details the architectural changes and new technologies implemented to enable WhatsApp's multi-device capability. Key contributions include: rethinking the architecture to remove the smartphone as the sole source of truth, introducing individual identity keys for each device, developing a client-fanout approach for message encryption and transmission, extending security codes to represent all device identities, implementing Automatic Device Verification, and providing users with control over linked devices. For voice/video calls, new methods for generating and distributing SRTP master secrets were developed. Message history synchronization involves initial encrypted bundle transfers and ongoing state synchronization via the server with end-to-end encrypted metadata.

Jun 2021Asicmon: A platform agnostic observability system for AI acceleratorsIntroduced Asicmon, a platform-agnostic observability system for AI accelerators, which abstracts custom interfaces and provides a standard interface for internal tools. Developed Asimov, a custom specification language that compiles to efficient C++ monitoring modules, simplifying the development and rapid prototyping of new accelerators. Introduced Atrace, an accelerator tracing solution for remote collection and analysis of traces on production servers.

Jun 2021Consolidating Facebook storage infrastructure with Tectonic file systemIntroduced Tectonic, a data center scale distributed file system designed to consolidate storage infrastructure. Tectonic scales to exabytes, disaggregates metadata into independently scalable layers and hash-partitions them into a shared key-value store, and uses a linearly scalable storage node layer. It simplifies performance isolation by managing resources between traffic groups and enables tenant-specific optimizations through a client-driven microservice architecture. This consolidation reduced the number of data warehouse clusters by 10x, improving resource utilization and operational simplicity.

May 2021Running Border Gateway Protocol in large-scale data centersThis post details Meta's implementation of Border Gateway Protocol (BGP) for large-scale data center routing. It describes the scalable design, software implementation, and operations of their BGP-based routing design, which aims to build networks quickly and provide high availability. Key technical contributions include a uniform AS numbering scheme, hierarchical route summarization, integrated policy configuration for reliable communication and traffic diversion, and the development of an in-house BGP agent with a custom testing and incremental deployment framework to ensure fast, frequent changes to the network infrastructure.

Feb 2021Mitigating the effects of silent data corruption at scaleThis post details research into mitigating silent data corruption at scale, a widespread problem for large-scale infrastructure systems that can propagate across the stack and manifest as application-level problems, leading to data loss and requiring months to debug. It describes best practices for detecting and remediating silent data corruptions on a scale of hundreds of thousands of machines, including hardware resiliency, production detection mechanisms, and robust fault-tolerant software architectures. A real-world example of silent data corruption within a data center application, leading to missing rows in the database, is detailed, along with a high-level debug flow to determine root cause and triage the missing data.

Feb 2021FOQS: Scaling a distributed priority queueThis post introduces FOQS (Facebook Ordered Queueing Service), a new fully managed, horizontally scalable, multitenant, persistent distributed priority queue built on sharded MySQL. FOQS enables developers to decouple and scale microservices and distributed systems by providing a robust queueing mechanism with features for item prioritization, delayed delivery, and configurable delivery semantics (at-least-once/at-most-once). It details the item structure (namespace, topic, priority, payload, metadata, deliver_after, lease duration, unique ID, TTL), topic and namespace concepts, and the implementation details of Enqueue, Dequeue, Ack, and Nack operations. Key architectural components include Shard Manager for shard assignment, a Thrift API, circuit breaker design for shard health, and a Prefetch Buffer for efficient high-priority item retrieval during dequeues. The post also discusses the rationale for a pull-based interface over push for diverse workload needs.

Dec 2020On the value of diversified recommendationsThis post introduces the concept of diversified recommendations in recommendation systems, moving beyond purely predictive accuracy to consider factors like preference broadening. It uses Polya and balancing process models to illustrate self-reinforcing and self-balancing dynamics in user preferences. The post details the complexities of real-world preferences (multidimensional, soft, contextual, dynamic) and outlines practical methods for diversification, including author, media type, and semantic diversity. It also discusses exploring similar semantic nodes in content graphs, maintaining long and short-term preferences, utilizing explore-exploit tradeoff algorithms (Epsilon-greedy, Upper confidence bound), qualitative trade-off computation, and negative guardrails based on user feedback. The core technical contribution lies in proposing a framework for balancing engagement and diversity in recommendation systems by understanding and addressing the nuances of user preferences.

Dec 2020How Instagram suggests new contentThis post introduces 'Suggested Posts' in Instagram, a feature designed to blend the familiarity of the home feed with the exploration of new content. It details a two-stage recommendation system: candidate generation (using embeddings and co-occurrence for implicit interest discovery) and candidate selection (employing a user value model with ML algorithms like MTML, GBDT, and LambdaRank). The system addresses the cold-start problem with fallback graph exploration and popular media. A key design principle, 'Feels Like Home,' is implemented through source prioritization (Home >> Other Surfaces > Fallback), consistent freshness heuristics, and similar media type distributions. The post also outlines a query language for rapid prototyping of sourcing queries and discusses the use of various features and model tuning strategies.

Nov 2020FioSynth: A representative I/O benchmark and data visualizer for data center workloadsIntroduced FioSynth, an open-source tool for automating storage workload execution and parsing results. FioSynth synthesizes I/O traces from production services to create rate-limited workloads that focus on outlier latency, a critical factor for Facebook applications. It includes dozens of workload suites, scalable capacity adjustments, CSV summarization, client/server mode for parallel execution, optional device health logging for write amplification factor calculation, and preconditioning/measurement phases. FioSynth has been used internally to identify performance inefficiencies and reproduce production issues, and is now available to the open-source community to aid storage device manufacturers and standardize flash preconditioning.

Nov 2020Building a ubiquitous shared infrastructure using TwineThis post details the evolution and impact of Twine, Meta's cluster management system, in achieving a ubiquitous shared infrastructure. Key contributions include scaling a single control plane to manage one million machines per region, supporting workload-specific customization via host profiles, and introducing the TaskControl API for collaborative workload life cycle management. The adoption of smaller, power-efficient machines has led to significant power and cost savings. The post highlights the shift from siloed machine pools to a unified infrastructure, with twshared hosting 56% of the fleet and projected to cover all compute services by early 2022.

Oct 2020How we scale Live streaming for millions of viewers simultaneouslyThis post details the evolution of Meta's Live streaming infrastructure to support millions of concurrent viewers for both UGC and professional broadcasts. Key contributions include: building new ingestion capabilities with redundancy (managed encoders, dedicated connections, failover mechanisms) to bridge traditional TV production and internet-based ingestion; scaling delivery infrastructure to manage massive fanout and mitigate the 'thundering herd' problem through novel request coalescing and cache sharding; and optimizing last-mile connectivity by increasing local interconnection capacity with ISPs and exploring LTE-Broadcast. The post also highlights the discovery of new failure modes and the importance of addressing them for large-scale events.

Oct 2020Introducing resctl-demo: Better resource control with simulationIntroduces resctl-demo, a tool for simulating system resource conflicts to test and develop strategies for resource control and optimization in server deployments. It provides an AMI and USB drive image for easy deployment and aims to improve reliability, responsiveness, and utilization by allowing developers to build intuitive working knowledge of resource control.

Sep 2020Throughput autoscaling: Dynamic sizing for Facebook.comThis post details the implementation of throughput autoscaling for Facebook.com's web tier. It describes the transition from static capacity provisioning to dynamic sizing based on workload-driven throughput metrics. The system models predicted steady-state and disaster demand using ML and estimates supply through internal load testing. This approach optimizes resource utilization by dynamically adjusting capacity to meet fluctuating global traffic patterns and disaster readiness requirements.

Sep 2020Fault tolerance through optimal workload placementThis post details Meta's evolution in fault tolerance through optimal workload placement across data center fault domains. It introduces the concept of fault domains and the need for buffer capacity to mitigate sub-data center failures. The post outlines a multi-level approach to achieve optimal placement: hardware placement (using integer linear programming and considering constraints like changing demand and evolving data center designs), service placement (addressing specific hardware requirements and service affinities), and continuous rebalancing. It also discusses strategies for handling existing capacity through physical rack moves. The core technical contribution is the systematic approach to distributing hardware, services, and data to minimize the impact of localized failures and improve overall system resilience.

Aug 2020Containerizing ZooKeeper with Twine: Powering container orchestration from withinThis post details the migration of all ZooKeeper deployments to Meta's internal container orchestrator, Twine. It describes the challenges of running a stateful, quorum-based system like ZooKeeper on a platform designed for stateless services, particularly the Twine Shared Pool. Key technical contributions include the development of a 'task controller' to enforce ZooKeeper's durability and quorum requirements on Twine's scheduler, and the engineering of 'break-glass' recovery techniques to mitigate circular dependencies with other Twine-powered infrastructure services (e.g., service discovery, configerator, fbpkg). The post highlights the benefits of this migration, including reduced operational toil, improved reliability through leveraging standard infrastructure features, and the elimination of years of obsolete custom code.

Aug 2020Scaling services with Shard ManagerThis post introduces Shard Manager, a generic platform developed at Meta to address the problem of duplicated custom sharding solutions across teams. It details the concept of sharding, contrasting it with hashing, and explains the need for shard management capabilities like failover, load balancing, and shard scaling. The post categorizes sharded applications into three types (primary only, secondaries only, primary-secondaries) and outlines the three-step process for building applications with Shard Manager: implementing a shard state transition interface, providing intent-based specifications for constraints, and using a common routing library. It highlights Shard Manager's wide adoption, managing tens of millions of shards on hundreds of thousands of servers across hundreds of applications, and its integration into the overall infrastructure ecosystem.

Aug 2020Asynchronous computing @Facebook: Driving efficiency and developer productivity at Facebook scaleThis post details the evolution of Meta's Async system for handling asynchronous requests. The original system used a centralized database with simple prioritization, which failed to scale. The reimagined Async system addresses scalability challenges by introducing: 1. Delay Tolerance: Prioritizing requests based on how long they can be delayed. 2. Capacity Optimization: Utilizing queueing (multiple queues ordered by deadline), time shifting (predictive and deferred compute), and batching to spread workload and reduce pressure on components. 3. Capacity Regulation: Implementing policies to prevent any single use case from consuming excessive resources. The system now uses a multitenant architecture to maximize server utilization.

Aug 2020Pcicrawler: A Python-based command-line interface tool to debug PCI issues at scaleIntroduced pcicrawler, a Python-based command-line interface tool for debugging PCI/PCIe issues at scale. It provides detailed hardware information, visualizes PCI topology, and offers machine-parsable JSON output for automation, improving upon legacy tools like lspci. It is designed to be lightweight, efficient, reliable, and safe for production environments.

May 2020Accelerometer and SoftSKU: Improving hardware platform performance for diverse microservicesIntroduced SoftSKU, a strategy to tailor existing server processors to specific microservices using OS and hardware configuration knobs, achieving up to 7.2% performance improvement without new hardware. Developed Accelerometer, an analytical model to predict hardware acceleration gains for data center microservices with less than 3.7% error rate, enabling informed hardware investment decisions.

May 2020Rebuilding our tech stack for the new Facebook.comThis post details the rearchitecture of Facebook.com from a server-rendered PHP site to a client-driven React and Relay application. It introduces techniques for improving performance and maintainability, including: generating atomic CSS to reduce bundle size by 80%, collocating styles with components, using CSS variables for theming (dark mode), inlining SVGs for faster rendering, and implementing JavaScript code-splitting with three loading tiers (Tier 1 for initial paint, Tier 2 for full above-the-fold content, and Tier 3 for post-display functionality). The post also discusses the adoption of React and Relay, and the technical mantras guiding the rewrite: 'As little as possible, as early as possible' and 'Engineering experience in service of user experience'.

Apr 2020Turbine: Facebook’s service management platform for stream processingIntroduced Turbine, a service management platform for Facebook's stream processing, designed to bridge the gap between general-purpose cluster management frameworks and stream processing requirements. Turbine features a fast and scalable task scheduler, an efficient predictive autoscaler, and an atomic, consistent, isolated, durable, and fault-tolerant application update mechanism. It manages tens of thousands of machines and thousands of streaming pipelines, processing hundreds of gigabytes of data per second in real time, enabling strict SLOs like 90-second end-to-end latency guarantees.

Feb 2020ZooKeeper Meetup@Facebook: Advancing the state of distributed coordinationThis post details improvements to ZooKeeper's write throughput (up to 10x) through changes to the snapshot scheduler and the addition of batch commits. It also describes a new lock API for semaphore optimization that reduces server round trips and QPS, demand control features for multitenancy (data size and request rate quotas) to mitigate noisy neighbor situations, and the migration of the entire ZooKeeper fleet to an in-house containerization system, replacing bare-metal deployments.

Dec 2019Systems @Scale Tel Aviv 2019 recapThis post provides a recap of the Systems @Scale Tel Aviv 2019 conference, featuring technical discussions on scaling Facebook's data center infrastructure, improving unit test quality through mutation testing, managing trade-offs for data prefetching in Facebook Lite, adapting system designs to evolving hardware technologies, developing a graph-based ORM for Go, analyzing memory at scale for memory-bound Java services, aligning data points from diverse sources in real-time, managing monorepos for large codebases, operating low-latency fraud prevention systems, and detecting significant metric movements at scale through an alerting framework.

Oct 2019Simon Marlow, Simon Peyton Jones, and Satnam Singh win Most Influential ICFP Paper AwardThis post details the award-winning work on the "Runtime Support for Multicore Haskell" paper, which significantly improved Haskell's performance on parallel hardware by addressing bottlenecks in the GHC runtime system. This resulted in up to 8x speedups for benchmarked programs, making Haskell suitable for large-scale production systems like Facebook's abuse protection system, Sigma. The work has also influenced systems in pharmaceuticals, finance, and biotech, and contributes to ongoing efforts in source code analysis.

Oct 2019Scribe: Transporting petabytes per hour via a distributed, buffered queueing systemThis post details the architectural revamp and simplification of Scribe, Meta's distributed, buffered queueing system designed to transport petabytes of log data per hour. It introduces the current design, including Producer libraries, the Scribed local daemon, the Write Service for batching and forwarding logs to storage back-ends (now LogDevice, previously HDFS), and the Read Service for streaming log access. Key design decisions emphasize simplicity, availability, scalability, and multitenancy, with a focus on the data plane components and the trade-offs made for high-throughput, low-latency offering.

Sep 2019MaRS: How Facebook keeps maps current and accurateIntroduced Logical Changesets (LoChas) and a machine-augmented automatic review system (MaRS) to improve the ingestion and accuracy of OpenStreetMap (OSM) data for Meta's applications. LoChas break down OSM changesets into individual CRUD operations and cluster them for efficient human review. MaRS uses heuristics and machine learning to automate the evaluation of LoChas, reducing the need for manual intervention. This system ensures that map data remains fresh and accurate by processing changesets efficiently and maintaining geometric validity.

Aug 2019Redesigning our systems to provide more control over Off-Facebook activityRedesigned systems to store and process Off-Facebook activity, introducing a separable identifier (SID) system to replace User IDs (UIDs) for disconnecting data. Developed a Measurement ID (MID) for aggregated measurement reporting and implemented encrypted identifiers (EIDs) for compound data types. Addressed special cases for security, safety, and user experience, including password protection for detailed activity lists and not displaying activity when user authentication is uncertain. Modified data warehouse tables to separate on-site and off-site data.

Aug 2019Performance @Scale 2019 recapThis post summarizes presentations from the @Scale 2019 conference, highlighting advancements in performance engineering across various domains. Key contributions include methodologies for analyzing Facebook AI workloads on accelerated platforms (TPUs/GPUs), techniques for scaling ML models, and strategies for improving mobile app performance (app size, startup, crash rates). It also covers open-source browser contributions to enhance web app performance and the development of tools like FlameScope for detailed profiler analysis. Furthermore, it details efforts in monitoring real user perceived performance on native apps and scalable approaches to generating iOS binary order files for improved startup performance.

Aug 2019Systems @Scale 2019 recapThis post is a recap of the Systems @Scale 2019 conference, featuring presentations on various large-scale systems engineering topics. Key contributions include discussions on Apache ZooKeeper's genesis, Apache Hive's evolution to enterprise-grade warehousing, Delos as a storage system for Facebook's control plane, Accordion for memory organization in LSM key-value stores, distributed tracing systems (Jaeger and Canopy) for observability, continuous deployment at Facebook scale, Affirm's observability infrastructure, LinkedIn's PYMK recommendation service and its reliance on Venice, Facebook's cluster management system Tupperware, preemption algorithms in Nomad, and disaster recovery strategies at Facebook scale. The post highlights trade-offs in system design, performance optimizations, and lessons learned from building and operating these systems.

Jul 2019OIL+VCache: File abstraction for distributed systemsThis post introduces OIL (Output-Input Language) and OIL+VCache. OIL provides a unified namespace and abstraction for files across heterogeneous storage systems, allowing developers to configure I/O flows via Directed Acyclic Graphs (DAGs) to achieve desired trade-offs (e.g., consistency vs. latency) on a per-file or per-I/O basis. It expands POSIX APIs to better capture distributed system nuances, including blocking reads beyond EOF for efficient communication. VCache is a distributed virtual memory system integrated with OIL, offering performance gains through caching layers and supporting various write modes. OIL+VCache enables flexible composition of storage, optimized routing, and control over replication and quorum, with separate DAGs for data and metadata.

Jun 2019Delos: Simple, flexible storage for the Facebook control planeIntroduced Delos, a new architecture for replicated storage systems designed for control plane services. Delos features a modular, layered design with pluggable APIs and ordering mechanisms (SimpleLogs) that dynamically stitch together via a VirtualLog. This allows for rapid deployment (8 months for initial version) and evolution, demonstrated by a 10x latency reduction achieved by hot-swapping a new ordering mechanism without downtime. Delos separates API materialization from update ordering and reconfiguration, enabling flexibility for diverse storage needs like key-value, hierarchical namespaces, and queues, while ensuring low dependency and strong consistency, durability, and high availability.

Jun 2019Efficient, reliable cluster management at scale with TwineThis post details the evolution and architecture of Twine, Meta's cluster management system that orchestrates containers on millions of servers. It highlights key features such as seamless support for stateful services via the TaskControl interface, a single control plane across data centers using Resource Broker for dynamic server-scheduler binding and maintenance coordination, and transparent control plane sharding for scalability. The post also discusses the shift towards a global shared fleet and the challenges of managing competing requirements, emphasizing Twine's role in enabling elastic compute and efficient capacity shifting.

May 2019Extending DHCPLB: The path from load balancer to serverThis post details the evolution of Meta's DHCP infrastructure, moving from ISC-DHCP to ISC Kea, and finally to an extended DHCPLB that functions as a server. The primary technical contribution is the replacement of the single-threaded Kea DHCP server with a multithreaded DHCPLB server, built upon a new Go DHCP library. This change addresses performance bottlenecks caused by Kea's single-threaded nature and the blocking Recv-Q buffer, leading to significantly improved throughput, reduced server count (10x fewer servers for the same traffic volume), and faster iteration cycles. The post also discusses the challenges of previous architectures, the benefits of the new Go DHCP library, and the use of DHCPLB for A/B testing and phased rollouts.

Apr 2019Open-sourcing F14 for faster, more memory-efficient hash tablesThis post introduces F14, a new 14-way probing hash table implementation within the Folly library. F14 aims to provide a superior default hash table by balancing performance and memory efficiency. Key technical contributions include: 1) Exploiting vector instructions (SSE2/NEON) for parallel intra-chunk searching, reducing collision impact on CPU pipelining. 2) A chunking strategy that maps keys to blocks of slots, significantly reducing collision probability. 3) A novel reference-counted tombstone mechanism using an auxiliary overflow bit to efficiently handle erasures and improve probe lengths. 4) Multiple memory layouts (F14Node, F14Value, F14Vector) to optimize for different use cases, particularly improving memory utilization with a high load factor (12/14) in F14Value. F14 simplifies the choice of hash table by offering two primary variants based on reference stability needs.

Mar 2019Sharing a common form factor for accelerator modulesIntroduced the Open Compute Project (OCP) Accelerator Module (OAM) as a common form factor specification for AI hardware accelerators. This addresses the challenges of diverse and unoptimized existing form factors (like PCIe CEM) by providing a standardized module with defined power, dimensions, and high-speed interconnect capabilities to enable flexible and scalable multi-ASIC solutions for AI workloads.

Mar 2019Open-sourcing homomorphic hashing to secure update propagationThis post introduces and open-sources LtHash, a specific homomorphic hashing algorithm implemented for secure update propagation in Meta's Location Aware Distribution configuration management system. It details the challenges of securing update propagation at scale, including issues with signing each update or signing the entire database. LtHash is presented as an efficiently updatable hashing solution that allows for the computation of updated database hashes without recomputing from scratch, leveraging set homomorphism and collision resistance. The implementation is released in the Folly C++ library.

Dec 2018HyperLogLog in Presto: A significantly faster way to handle cardinality estimationThis post details the implementation of the HyperLogLog (HLL) algorithm in Presto, a distributed SQL query engine. It explains the evolution of cardinality estimation algorithms from simple estimators to probabilistic counting, LogLog, and finally HLL. The post highlights the performance improvements achieved (7x to 1,000x speedup) and the significant reduction in memory usage (from terabytes to less than 1 MB) for tasks like counting distinct users. It also introduces newly open-sourced functions for further computational savings.

Nov 2018Getafix: How Facebook tools learn to fix bugs automaticallyThis post introduces Getafix, a novel tool that automatically finds and suggests fixes for software bugs by learning from past engineer code changes. It enhances code quality and system stability by integrating with existing testing tools like Sapienz and Infer, and represents a significant advancement in automated bug remediation at scale.

Oct 2018Facebook open-sources new suite of Linux kernel components and toolsThis post introduces a suite of open-source Linux kernel components and tools developed at Facebook to address critical fleet management issues in large-scale cloud computing. Key contributions include: BPF for safe and flexible kernel code execution (used in Katran load balancer), Btrfs as a next-generation filesystem for fault tolerance and administration (improving I/O control with cgroup2), Netconsd for lightweight UDP-based netconsole logging, Cgroup2 for advanced workload grouping and resource control (driving multi-tenancy and isolation), PSI for quantifying resource shortages (memory, CPU, I/O) to enable proactive responses, and Oomd as a userspace OOM killer with greater context awareness. These components are in production across Facebook's infrastructure and are now available to the open-source community.

Oct 2018MobileLab: Highly accurate testing to prevent mobile performance regressionsThis post introduces MobileLab, a new system for highly accurate mobile performance testing designed to prevent regressions as small as 1 percent. It details the development of a validation framework, strategies for eliminating variance through consistent device and application performance (CPU/GPU frequency locking, HTTP proxy for server isolation, application state backup/restore), and further optimizations like using RAM disks for disk I/O, resetting device clocks, dismissing crash dialogs, and establishing direct socket connections for metric reporting. These efforts reduced variance by an order of magnitude, enabling reliable detection of sub-1 percent regressions with only 50 trials.

Oct 2018Managing data store locality at scale with AkkioIntroduced Akkio, a new data placement service (DPS) that optimizes data locality for geo-distributed data stores at Facebook's scale. Akkio splits data into microshards and intelligently moves them to regions close to typical access patterns, reducing duplication, WAN traffic, and perceived latency by up to 50%. It addresses the limitations of traditional distributed caches for Facebook's workloads by handling high read-write ratios and strong consistency requirements. The system comprises a metadata tier for microshard-to-placement mapping, an Access DB for storing historical access patterns, and a Data Placement Service (DPS) that uses policies and capacity information to trigger data migrations. Akkio operates at microshard granularity and supports both strongly and eventually consistent backends.

Oct 2018Under the hood: Facebook Marketplace powered by artificial intelligenceThis post details the AI-powered systems behind Facebook Marketplace, focusing on product indexing, retrieval, and seller assistance. Key technical contributions include: a multimodal ranking system combining Lumos (image understanding) and DeepText (text understanding) for product indexing; the use of FAISS for efficient similarity search to reduce computing load and speed up retrieval; distributed training of neural network models across multiple machines to reduce training time from over a week to 1-2 days; optimizations to increase QPS and CPU utilization by tuning classification threads and improving memory purging configurations (MALLOC_CONF='opt.decay_time:-1'); AI-driven autosuggestions for seller categories and pricing; and the use of M (an AI assistant) in Messenger for automated suggestions, including replies, product recommendations, and message translations via neural machine translation. The post also mentions ongoing testing of image-based search for visually similar products.

Sep 2018Open-sourcing LogDevice: A distributed data store for sequential dataThis post announces the open-sourcing of LogDevice, a distributed data store for sequential data developed and deployed within Facebook. LogDevice provides durability, high availability, and total order of records under failures, making it suitable for event streaming, replication pipelines, transaction logs, and deferred work journals. It includes a command-line administration tool, LDShell. The release aims to foster community contributions, particularly in integration with cloud computing environments, monitoring systems, and configuration management.

Jul 2018Systems @Scale 2018 recapThis post is a recap of the first-ever Systems @Scale conference, featuring talks on managing large-scale information systems. Key contributions include Jay Parikh's keynote on Facebook's speed and scale improvements, Doug Terry's talk on DynamoDB Global Tables geo-replication, Facebook engineers' discussion on systemd for containers and composable services, Shopify's approach to Kubernetes application migrations for stateful applications, Liz Fong-Jones's insights on improving debugging strategies for faster outage resolution, Facebook engineers' presentation on Location-Aware Distribution (LAD) for data distribution, Lyft's adoption of Envoy for service mesh, Oath's modernization of Yahoo Mail, Facebook's Web@FB continuous and reliable release process, and Facebook's vision for a global shared fleet for Compute-as-a-Service.

Jul 2018Open-sourcing oomd, a new approach to handling OOMsThis post introduces oomd, a new userspace out-of-memory (OOM) killer developed by Facebook. It addresses limitations of the traditional Linux OOM killer by providing faster, more reliable, and customizable OOM handling. Key contributions include the use of Pressure Stall Information (PSI) and cgroup2 for proactive monitoring and a plugin system for custom OOM response strategies, significantly reducing livelocks in large-scale infrastructure.

Jul 2018Location-Aware Distribution: Configuring servers at scaleThis post introduces Location-Aware Distribution (LAD), a new peer-to-peer system designed to address the scalability challenges of distributing configuration changes to millions of servers. LAD improves upon the previous ZooKeeper-based system by supporting much larger configuration files (100 MB vs 5 MB), increasing the number of subscribers per distributor (40,000 vs 2,500), and separating data storage from the distribution framework. It utilizes a tree-based peer-to-peer network with sharding to manage updates efficiently and reduce latency, while also highlighting lessons learned in tooling, monitoring, and failure testing for large-scale distributed systems.

Jul 2018XARs: An efficient system for self-contained executablesIntroduced XARs (Executable Archives) as a system for distributing self-contained executables that encapsulate data and code dependencies. XARs leverage a FUSE-based filesystem and squashfs for on-demand decompression and efficient caching, aiming to improve deployment speed, robustness, and efficiency compared to previous methods like PARs and PEX files. Benchmarks show improvements in file size and start times, particularly for short-running executables. XARs are open-sourced via GitHub and PyPI.

Jun 2018Spiral: Self-tuning services via real-time machine learningIntroduced Spiral, a system for self-tuning high-performance infrastructure services using real-time machine learning. Spiral replaces hand-tuned heuristics with ML classifiers, enabling services to adapt to changing conditions and optimize in minutes rather than weeks. It integrates via prediction and feedback call sites and can operate in embedded or backend modes. A case study demonstrates its application in automating reactive caching heuristics by classifying database updates as spam or ham to improve efficiency.

Jun 2018Accelerate large-scale applications with BOLTIntroduced BOLT, a binary optimization and layout tool that addresses instruction starvation by optimizing instruction placement in memory. BOLT reconstructs control flow graphs, uses LLVM's MCInst format, and supports Linux binaries (x86-64, AArch64). It leverages sample-based profiling from Linux `perf` tool. BOLT achieved 3-8% performance improvement on HHVM and 2-15% for other services, by isolating frequently executed code and optimizing for hardware features like macro-fusion. BOLT also performs advanced jump table placement, identical code folding, PLT optimization, and constant load elimination.

Jun 2018Building data science teams to have an impact at scaleThis post details the integration of data science teams with infrastructure engineering teams at Meta to tackle large-scale challenges. It highlights specific contributions in: 1. CDN cache efficiency through ML models predicting future content requests. 2. Performance regression detection tools for both client-side and server-side applications, using custom algorithms to account for trends, seasonality, and noise. 3. Data center demand forecasting using custom Bayesian time series and deep learning models, and predicting the impact of infrequent events on infrastructure.

Apr 2018AV1 beats x264 and libvpx-vp9 in practical use caseThis post details the evaluation of the AV1 video codec against x264 and libvpx-vp9 in practical Facebook video use cases. It presents experimental results showing AV1's superior compression efficiency (e.g., 50.3% gain over x264 main profile in PSNR for CRF/QP mode) but also highlights its significantly longer encoding times (e.g., 5721.5x increase over x264 main profile). The methodology involved testing with 400 top-viewed Facebook videos, decompressing and re-compressing them with AV1, x264, and libvpx-vp9 using both CRF/QP and ABR encoding modes. The results are presented using Bjontegaard-Delta rate (BD-rate) metrics for PSNR and SSIM, and encoding time increase ratios.

Mar 2018Fabric Aggregator: A flexible solution to our traffic demandThis post introduces the Fabric Aggregator, a novel distributed network system designed to scale data center network capacity. It details a disaggregated approach using simple, open building blocks like Wedge 100 switches and FBOSS, enabling flexible adaptation to varied traffic patterns and future growth. The implementation of a two-layer cross-connect architecture (downstream for east/west traffic and upstream for north/south traffic) allows for independent scaling of capacities. The post also discusses operational aspects, redundancy, and different deployment types (single-rack and multi-rack) with various cabling configurations, emphasizing efficiency, power savings, and increased flexibility.

Mar 2018Performance @Scale 2018 recapThis post is a recap of the Performance @Scale 2018 conference, highlighting presentations on distributed trace processing for performance regression detection (Execution Graphs at Microsoft), understanding app performance in the wild (Profilo at Facebook), robust anomaly detection for real user monitoring data (ThirdEye at LinkedIn), iOS VM and loader considerations, mobile performance testing in real user conditions (Headspin), global network emulation (Magic Modem at Netflix), scaling real-time infrastructure for global shopping holidays (Alibaba), and static resource delivery at Facebook. It serves as a summary of the state of performance engineering discussions and advancements presented at the conference.

Feb 2018How production engineers support global events on FacebookThis post details how Meta's Production Engineering team supports global events on Facebook, specifically focusing on the planning and infrastructure required for Facebook Live during New Year's Eve. It outlines the different types of load variance (routine, spontaneous, planned), the architecture of Facebook Live, key load metrics (total broadcasts, peak concurrent broadcasts, load on dependent systems), and the process of estimating and scaling for peak traffic. It also describes load testing methodologies (artificially increasing load, synthetic load tests, shadow traffic) to ensure system reliability during these high-traffic periods.

Jan 2018Android native library mergingThis post details a solution for merging multiple Android native libraries into a single shared object to overcome the pre-Android 4.3 native library limit. It describes integrating this capability into the Buck build system, handling duplicate JNI_OnLoad symbols using custom ELF sections and registration objects, and adapting the SoLoader library to manage merged library names. The solution also addresses challenges with weak symbols on older Android versions and linker symbol conflicts. A demonstration repository and analysis script are provided to illustrate the techniques and guide merging strategies.

Oct 2017Mobile @Scale 2017 recapThis post summarizes the Mobile@Scale 2017 conference, showcasing Meta's and other industry leaders' advancements in scaling mobile products and services. Key contributions include discussions on multi-modal interfaces, WhatsApp's infrastructure evolution within Facebook's data centers, multi-threaded rendering on Android using Litho & Infer, modern server-side rendering with React, real-time analytics for billions (Fabric), scaling mobile apps like Runkeeper and Instagram, Android development processes at Facebook, zero power voice activation, scaling Recommendations, high-performance app development and bottleneck analysis, battery instrumentation for Facebook apps, and travel tech at scale with Hopper. It reinforces the ongoing commitment to mobile scalability within the broader @Scale initiative.

Sep 2017Migrating a database from InnoDB to MyRocksThis post details the successful migration of Meta's UDB database tier from compressed InnoDB to MyRocks, a RocksDB storage engine for MySQL. The migration achieved a 50% reduction in storage usage by addressing InnoDB's space-bound limitations on Flash storage. Key technical contributions include the implementation of MyRocks features like clustered index, transaction support, and efficient deletions, as well as a robust migration strategy involving replication between InnoDB and MyRocks, comprehensive data consistency verification (primary/secondary key checks, cross-instance checks, shadow queries), and a phased rollout process. Lessons learned emphasize the importance of migration time in technology selection, understanding service usage for feature prioritization, starting with small read-only instances, and leveraging open-source collaboration.

Aug 2017LogDevice: a distributed data store for logsThis post introduces LogDevice, a new distributed data store specifically designed for logs. It addresses the challenges of achieving highly available and durable record storage with a repeatable total order on records at Facebook's scale. Key technical contributions include: decoupling record sequencing from storage, using non-deterministic record placement for improved write availability and load balancing, employing epochs for sequencer availability, and implementing many-to-many rebuilding for data durability. The design allows for tunable performance, availability, and latency requirements to accommodate diverse logging workloads.

Aug 2017FCR: Open source command runner for network devicesIntroduced FBNet Command Runner (FCR), an open-source command runner service with a Thrift interface for running commands on network devices. FCR abstracts vendor-specific details, handles multiple device vendors and addresses, and scales to tens of thousands of devices, simplifying network monitoring, provisioning, and deployment. It integrates with the Robotron system for active monitoring.

Jun 2017Data @Scale 2017 RecapThis post recaps the 3rd annual Data @Scale conference, highlighting discussions on building large-scale storage systems and analytics, the impact of Big Data on machine learning, and the evolution of infrastructure and data center design. It features presentations on accelerating machine learning for computer vision, next-generation globally-distributed databases, real-time analytics DBMS, machine learning serving platforms, micro-service architectures, data warehousing and experimentation platforms, globally distributed data management systems, cloud specialization architectures, and bulk data movement for global data storage. The post also mentions a Women in Engineering Breakfast & Panel, emphasizing community and collaboration in addressing these challenges.

Jun 2017Dev Tools @Scale 2017 RecapThis post, a recap of the Dev Tools @Scale 2017 conference, highlights Meta's contributions and engagement in developing and utilizing advanced developer tools to enable large-scale development. It specifically showcases internal projects like Project OneWorld for resource management and testing, Infer for static code analysis, and Jupiter for job scheduling, alongside industry-wide discussions on code search, source control scaling, and build systems.

May 2017Managing resources for large-scale testingThis post introduces One World, a new unified resource management system for large-scale testing at Facebook. It details the architecture, including runtime worker services, the One World daemon, and the Jupiter scheduler. It explains how One World manages tens of thousands of resources (mobile devices, browsers, emulators) and executes over a million jobs daily. The post highlights the challenges of device reliability and the solutions implemented, such as state machines for worker services, health checks, and state restoration. It also describes the mechanisms for remote access to Android (via adb tunnels) and iOS (via FBSimulatorControl and WebDriverAgent) devices, and the role of the One World daemon in session management and resource leasing. The satellite mode for connecting local devices to the global system is also presented.

May 2017Jupiter: A high-performance job-matching serviceThis post introduces Jupiter, a high-performance job-matching service designed to alleviate bottlenecks in Facebook's engineering workflow. Jupiter efficiently matches jobs to workers based on multidimensional capabilities, ensuring atomic job acquisition and supporting flexible ordering policies. It is horizontally scalable, written in C++, and accessible via Thrift, handling hundreds of thousands of requests per second per shard with tens of microseconds latency for work acquisition. Jupiter decouples job matching from persistence, enhancing reliability and specialization.

May 2017OnlineSchemaChange rebuilt in PythonThis post announces the rebuilding of OnlineSchemaChange (OSC) in Python, improving upon the original PHP version. Key contributions include a standalone CLI for easier use, a more flexible architecture for feature development and testing, enhanced reliability with data consistency checks, and support for deep integration within Python-based infrastructures via a portable module and plugin system. Future plans include triggerless schema changes and utilizing native online DDL.

Apr 2017Relay Modern: Simpler, faster, more extensibleIntroduced Relay Modern, a new version of the Relay framework. Key improvements include: simplified API, enhanced performance through static queries and ahead-of-time optimization, simpler mutations with explicit field fetching and cache update specifications, and built-in garbage collection for cache eviction. The framework is now modularized into a compiler, runtime, and React integration layer, increasing extensibility. An iterative adoption path with a compatibility API is provided for existing Relay applications.

Mar 2017Faiss: A library for efficient similarity searchIntroduced Faiss, a library for efficient similarity search on billion-scale datasets. Faiss provides state-of-the-art nearest-neighbor search implementations that are significantly faster than previous methods, with a focus on memory usage and speed. It includes GPU implementations for accelerated performance and supports various indexing methods to trade off accuracy for speed. The post details evaluation metrics (speed, memory usage, accuracy) and demonstrates performance on a 1-billion vector dataset, comparing it against existing research.

Mar 2017Evaluating boosted decision trees for billions of usersThis post introduces optimizations for gradient-boosted decision trees (GBDTs) in C++ to improve evaluation efficiency for large-scale ranking systems. It compares naive flat tree implementations with compiled trees, leveraging LIKELY/UNLIKELY annotations for branch prediction. Further improvements include range-based evaluation to optimize CPU cache usage by evaluating samples across tree ranges, and techniques for handling common and categorical features to reduce model size and evaluation time. Computational results show up to a 5x performance improvement over the flat tree implementation.

Mar 2017Introducing Bryce Canyon: Our next-generation storage platformIntroduced Bryce Canyon, a next-generation high-density storage platform designed for photos and videos, offering 20% higher HDD density than Open Vault. It features improved thermal and power efficiency, modularity for future growth, and flexible configurations (72-drive server, dual 36-drive servers, or 36/72-drive JBOD). It supports single or dual Mono Lake CPU modules for optimized performance per watt and integrates with OpenBMC for system management. The toolless design simplifies servicing and drive replacement.

Mar 2017The software ecosystem for storage hardwareThis post details the validation of software-defined storage solutions on open hardware (Leopard servers, Open Vault JBODs) within Meta's Disaggregate Lab. It highlights the validation of NetApp's ONTAP Select, StorageGRID, and AltaVault; IBM's Spectrum Scale; Excelero's NVME software; Weka.IO's flash-centric file system; Hedvig's unified storage platform; and Red Hat's GlusterFS and Ceph. The post emphasizes the growing software ecosystem for open hardware and encourages vendors and adopters to leverage the lab.

Feb 2017Using Apache Spark for large-scale language model trainingThis post details the migration of a large-scale, complex (100+ stage) N-gram language model training pipeline from HQL over Hive to Apache Spark. The primary technical contributions include: 1. Leveraging Spark's DSL and RDD interface to control data distribution and manage shard generation, enabling modularity and maintainability over monolithic SQL queries. 2. Invoking existing C++ binaries for language model training algorithms via Spark's `pipe()` operator, facilitating a smooth transition without rewriting core logic. 3. Addressing scalability challenges related to data skew in smoothing and sharding by implementing a progressive sharding approach (iterative sharding based on N-gram history length) and dynamically adjusting shard sizes to ensure balanced data distribution and job stability. This resulted in reduced resource usage and data landing time.

Feb 2017Beringei: A high-performance time series storage engineThis post introduces Beringei, a high-performance, in-memory time series storage engine developed to address the scaling limitations of Facebook's previous HBase-backed TSDB for monitoring infrastructure. Beringei is optimized for time series data, featuring a high write rate, low read latency, and efficient RAM usage. It employs a lossless streaming compression algorithm, specifically delta-of-delta encoding for timestamps and XOR-based compression for values, achieving over 90% compression. The system offers both a sharded service implementation with Grafana integration and an embedded library option. Beringei currently stores 10 billion unique time series and serves 18 million queries per minute, enabling real-time exploration and debugging.

Jan 2017The growing ecosystem around open networking hardwareThis post details Meta's contributions to and the growing ecosystem around open networking hardware, including the open-sourcing of Wedge 100, Backpack, and Voyager. It highlights collaborations with companies like Cumulus Networks, Big Switch Networks, Barefoot Networks, SnapRoute, Canonical, Cavium, and Apstra to integrate various network operating systems (Cumulus Linux, ONL, OpenNSL, FBOSS, SONiC, Ubuntu Core, FlexSwitch) and software components with this open hardware. It also announces the commercial availability of Wedge 100 and Backpack, and the development of new hardware variations like Wedge 100B and Wedge 100C.

Dec 2016Made in NY: The engineering behind social recommendationsThis post details the engineering behind Facebook's 'Recommendations' feature, which extracts place suggestions from free-text comments. It describes the use of conversational understanding (CU) and local search technologies to identify and resolve place entities from unstructured text. The post highlights the technical challenges of dynamically updating feed stories across platforms (Android and iOS) with place cards and map pins, including manual client-side caching strategies and the use of ComponentKit for dynamic UI rendering on iOS. It also outlines the tech stack for entity retrieval and scoring, involving Gaussian mixture models and LambdaMART algorithms.

Oct 2016Networking @Scale BostonThis post documents the Networking @Scale Boston conference, highlighting discussions on cloud networking, traffic configurations, content delivery network infrastructure, and the scaling of applications and services. Key contributions include insights into Facebook's Open/R routing platform, scaling Facebook Live, traffic configuration evolution, and securing network pipes. It also features discussions from other industry leaders on Google's networking efforts, Akamai's cloud networking, browser networking performance, and QUIC protocol adoption.

Oct 2016A comparison of state-of-the-art graph processing systemsThis post provides a quantitative and qualitative comparison between Apache Giraph and GraphX for large-scale graph processing. It details experimental methodology, including algorithms (PageRank, Connected Components, Triangle Counting), data sets (Twitter, UK web graph), and system configurations (workers, threads, GC). Key findings indicate Giraph's superior performance and scalability, processing significantly larger graphs with lower memory usage and machine hours compared to GraphX. While GraphX offers easier development for preprocessing tasks, Giraph is deemed better for production-scale workloads.

Sep 2016DHCPLB: An open source load balancerThis post details the development and productionization of DHCPLB, an open-source DHCP load balancer designed to address traffic imbalance issues in Facebook's production DHCP infrastructure. It covers the identification of the problem (uneven traffic distribution due to anycast and network topology), the hackathon proof-of-concept, and the intern-led project that added DHCPv6 support, A/B testing capabilities for KEA servers, MAC address-based overrides, integration with monitoring infrastructure, hot-reloading configuration, BGP advertisement via ExaBGP, and two balancing algorithms (RoundRobin and ModuloN). The post also outlines the architecture and deployment of DHCPLB within Facebook's data centers.

Sep 2016Facebook announces new tech at @Scale 2016This post announces new technologies presented at the @Scale 2016 conference, highlighting advancements in data compression (Zstandard), database storage engines (MyRocks), 360 video stabilization, and developer tools for automated regression detection (BrowserLab) and static analysis (Infer). These contributions showcase Meta's ongoing efforts to address large-scale engineering challenges and improve efficiency across various domains.

Aug 2016BrowserLab: Automated regression detection for the webThis post introduces BrowserLab, a system designed to automatically detect and prevent client-side performance regressions in web applications. It details the challenges of measuring client performance in a highly interactive web environment, the sources of noise in performance testing (environment, data, code non-determinism), and the system's design to mitigate these issues through HTTP proxies, backend proxies, and deterministic code execution. BrowserLab's ability to detect regressions as small as 20ms and its integration into the continuous integration pipeline for automated bisecting and alerting are highlighted as key contributions to maintaining site performance at scale.

Aug 2016Apache Spark @Scale: A 60 TB+ production use caseThis post details the migration of a 60 TB+ Hive workload for feature preparation in entity ranking to Apache Spark. It describes the challenges of the previous sharded Hive implementation and the process of scaling Spark to handle the large shuffle data size. Key contributions include reliability fixes for node reboots and unresponsive drivers, performance improvements through memory leak fixes in the sorter and Snappy optimizations, and the development of tools like Linux Perf/Flame Graph support for aggregated CPU profiling. Many of these improvements were contributed back to the Apache Spark open-source project.

Aug 2016MyRocks: A space- and write-optimized MySQL databaseThis post introduces MyRocks, a new open-source MySQL storage engine that integrates RocksDB as its backend. MyRocks addresses inefficiencies in InnoDB for space and write amplification on flash storage by leveraging RocksDB's compression efficiency, append-only LSM-tree structure, and optimized metadata handling. Benchmarks show MyRocks is significantly smaller and has lower write amplification than compressed InnoDB. The post also details the migration process from InnoDB to MyRocks in production, emphasizing zero downtime and data consistency verification.

Aug 2016Smaller and faster data compression with ZstandardIntroduced Zstandard (Zstd) 1.0, a new lossless compression algorithm and implementation designed for modern hardware. Zstd offers significant improvements over Deflate (gzip, zlib) in both compression and decompression speeds, while maintaining comparable or better compression ratios. Key technical innovations include a performance-first design optimized for modern CPUs, a scalable architecture with 22 compression levels, parallel execution capabilities through a new Huffman decoder (Huff0), and a branchless design to avoid costly pipeline flushes. Zstd's memory window is significantly larger than zlib's, allowing for better compression. The post details benchmark comparisons against lz4, xz, and zlib using the Silesia corpus and command-line tools, highlighting Zstd's superior trade-offs.

Aug 2016Robotron: Top-down network management at scaleThis post introduces Robotron, a top-down network management system designed to manage Facebook's massive production network of tens of thousands of devices and hundreds of thousands of servers. It details the system's evolution from initial monitoring efforts (FBNet) to a comprehensive solution covering network design, configuration generation, deployment, and monitoring. Key challenges addressed include distributed configurations, handling multiple domains, versioning, dependencies, and vendor differences. The system translates high-level operator intent into vendor-specific configurations and deploys/monitors them to ensure network state consistency. The post also highlights the open-sourcing of parts of Robotron, such as fbpush.

Aug 2016Inside Facebook’s hardware labs: Moving faster with more collaborationThis post details the evolution of Meta's hardware engineering practices, moving from isolated labs to a collaborative environment with the creation of 'Area 404'. It highlights the investment in state-of-the-art machinery for in-house modeling, prototyping, and failure analysis, significantly reducing development cycles. The post showcases how this collaborative approach accelerates innovation across infrastructure (e.g., Wedge, 6-pack, Open Vault), Connectivity Lab (e.g., Aquila, Terragraph), and VR (e.g., Surround 360, Oculus) projects.

Jul 2016The mobile device lab at the Prineville data centerThis post details the development and scaling of a mobile device lab within a data center to support large-scale on-device performance testing. It describes the evolution of hardware setups from 'sled' and 'gondola' to custom-built, EMI-shielded racks capable of holding 32 phones each, driven by Mac Minis or OCP Leopard servers. Software challenges addressed include the introduction of package management and Chef for automated configuration and monitoring of devices, ensuring uniformity and reducing host-related failures. The post also outlines future plans for densifying racks, improving Android testing infrastructure, extending Chef for iOS, and developing a generic platform for on-device testing using WebDriver.

Jun 2016Growing the Wedge/Wedge 100 communityThis post details the growth of the Open Compute Project (OCP) networking community, highlighting the development and adoption of open-source network switches like Wedge and Wedge 100. It showcases how disaggregating hardware and software in networking devices has enabled a flexible, scalable, and efficient ecosystem, with contributions from various companies and startups experimenting with and building upon these open platforms.

Jun 2016Data @Scale, June 2016 — RecapThis post recaps the Data @Scale conference in June 2016, showcasing discussions and presentations on exabyte-scale storage systems (Dropbox), integrated analytics for storage (Qumulo), MPP shared-nothing databases on flash (Presto Raptor by Facebook), lessons from studying modern databases and key-value stores (University of Wisconsin-Madison), flexible consistency levels in distributed key-value stores (Manhattan by Twitter), scale-out resource management (YARN++ by Microsoft), a unified model for batch and streaming data processing (Apache Beam by Google), data integration for visual analysis (Tableau), and disaggregated storage and compute for Map/Reduce (Facebook). It emphasizes the collaborative nature of the event and the sharing of practical insights for building and operating high-scale systems.

May 2016Python in production engineeringThis post details the extensive and foundational use of Python in Facebook's production engineering, highlighting its role in infrastructure management (hardware lifecycle, network setup, fault remediation, maintenance scheduling), platform services (job scheduling, binary distribution, TFTP, workflow execution), service configuration management (dynamic configuration generation), and operational efficiency (database health monitoring, phased rollouts). It quantifies Python's impact with statistics on codebase percentage, commits, and committers, and discusses the ongoing transition to Python 3 for performance gains.

Apr 2016Wangle — an asynchronous C++ networking and RPC libraryThis post introduces Wangle, a C++ library for asynchronous networking and RPC, heavily influenced by JVM libraries Netty and Finagle. It details Wangle's core components, including asynchronous building blocks from Folly (IO wrappers and Futures), concurrency primitives like CPU and IO thread pools with optimizations for different workloads, and a pipeline architecture for processing network data through a series of handlers and codecs. The post also covers bootstrapping servers and clients with ServerBootstrap and ClientBootstrap, and provides an example of building a file streaming server that leverages Wangle's capabilities to offload blocking disk reads to a CPU thread pool.

Apr 2016Facebook Chef cookbooksThis post announces the open-sourcing of 14 core Chef cookbooks developed and used internally at Meta. These cookbooks have been refactored to be generic and allow external users to adopt Meta's configuration management model. The post details the history of Meta's involvement with Chef and the motivation for releasing these cookbooks. It provides a quick start guide and demonstrates how to use the cookbooks with examples of setting sysctls, limits, logrotate, swap, modprobe, and fstab. It also explains the hierarchical configuration model enabled by these cookbooks and delves into the specifics of the `fb_fstab` cookbook, highlighting its advanced features for managing `/etc/fstab` and mounted filesystems.

Mar 2016Engineering for nostalgia: Building a personalized “On This Day” experienceThis post details the engineering efforts behind the "On This Day" feature's evolution, focusing on user experience research, filtering mechanisms, and a machine-learning-based ranking system. It highlights the use of automatic filters to exclude unwanted memories (e.g., with exes, blocked individuals) and the implementation of user preferences for blocklisting specific people or dates. The ranking model is described as being trained in real-time using personalization signals (user interactions, demographics, memory attributes) and content understanding from a computer vision platform that leverages deep convolutional neural networks to recognize visual concepts within photos and videos. This enables a more personalized and relevant memory surfacing experience.

Mar 2016Dragon: A distributed graph query engineIntroduced Dragon, a distributed graph query engine designed to optimize high-volume, multi-hop queries on the social graph. Dragon creates specialized indices on persistent storage (RocksDB) and employs socially aware inverted indices to improve data locality and reduce network transfer for graph traversal. It supports functional programming primitives for filtering and ordering graph edges, enabling efficient retrieval of ranked graph data.

Mar 2016Facebook’s new front-end server design delivers on performance without sucking up powerThis post details the redesign of Facebook's front-end web servers to improve performance per watt. Key contributions include the development of the Mono Lake (single-socket Xeon-D CPU with integrated chipset) and Yosemite (sled incorporating four Mono Lake boards with a multi-host NIC) server designs. This collaboration with Intel resulted in a 65W, single-socket CPU that doubles the number of CPUs per rack within the existing 11kW power budget, achieving a significant leap in performance cadence and efficiency compared to previous two-processor, higher-TDP designs. The design also addresses memory bandwidth limitations and explores LLC cache partitioning for future optimization.

Mar 2016Introducing Lightning: A flexible NVMe JBOFIntroduced Lightning, a flexible NVMe JBOF (Just a Bunch of Flash) designed for PCIe gen 3 end-to-end connectivity. Leveraged Open Vault (Knox) SAS JBOD infrastructure for faster time to market and modularity. Developed new boards: PCIe retimer card, PCIe Expansion Board (PEB) with PCIe switch and BMC, and PCIe Drive Plane Board (PDPB) with SSD connectors. Features include support for various SSD form factors (2.5", M.2, 3.5"), surprise hot-add/removal, OpenBMC management, multiple switch configurations for compute-to-storage ratio tuning, and support for up to four head nodes. Addressed design challenges in hot-plug, management, signal integrity, external PCIe cables, and power consumption (limiting to 14W per slot). Contributed the entire solution (hardware and software) to the OCP community.

Mar 2016How we built Facebook Lite for every Android phone and networkThis post details the architecture and engineering decisions behind Facebook Lite, a lightweight Android application designed for emerging markets. Key technical contributions include: a proxy server architecture with a thin client VM, server-side product code and resources to minimize APK size (<1MB), a custom message protocol over TLS for byte efficiency on 2G networks, dynamic shared dictionary compression (LZMA2 client-to-server, DEFLATE server-to-client), optimized image serving with exact-size and JPEG quality control, and a diff mechanism for screen updates to reduce data usage. The app was designed to run on Gingerbread and 2009-era devices, minimizing CPU, storage, and RAM usage on the client by offloading heavy tasks to the server.

Feb 2016Performance @Scale 2016 recapThis post recaps the Performance @Scale 2016 event, showcasing industry-wide discussions and advancements in making technology fast and efficient. It highlights specific techniques and tools for performance analysis and optimization, including BPF for Linux profiling, Facebook's web speed solutions (Time-to-Interact, Display Done, Early Flush, Big Pipe, Bootloader), automatic regression triaging (AutoTriage, Stack Trace Finder, Pushed Commit Search, Diff Ranker), Google's approach to ad revenue improvement through performance analysis, Microsoft's client-side performance investigation with PerfView, Google's high-performance networking in Chromium (histograms, QUIC protocol), and Facebook's mobile performance engineering with the Loom telemetry framework, as well as LinkedIn's Real User Monitoring (RUMv2) and visualization techniques.

Feb 2016Under the hood: Shipping #friendsday videosThis post details the engineering effort behind shipping personalized Friends Day videos to hundreds of millions of users. It covers load testing of the video generation pipeline, identifying and addressing bottlenecks to achieve significant processing time reduction (69%) through multithreading. It also describes capacity planning considerations including CPU, memory, storage IOPS, and network bandwidth. The post highlights the use of shared production infrastructure with automated capacity allocation and monitoring to maintain safety, and details the ranking system for photo selection, including the use of image-understanding AI. Finally, it outlines a staggered rollout plan to manage CDN load and user experience, and notes the use of React and React Native for rapid UI development.

Dec 2015Improving software RAID with a write-ahead logThis post details an improvement to software RAID 5/6 by introducing a write-ahead log. This addresses critical issues like the 'write hole' during power failures, significantly speeds up rebuild times after disk failures, and optimizes write performance by allowing aggregation of writes. The implementation uses an SSD as the log disk and has been merged into the Linux 4.4 kernel, enhancing the reliability of large-scale storage systems.

Nov 2015Optimizing Facebook for iOS start timeThis post details the extensive efforts undertaken to optimize the Facebook app's startup time on iOS. It outlines the process of defining stable performance metrics, distinguishing between cold and warm starts, and breaking down the cold start problem into request time, network time, and response processing time. Key innovations include UDP priming to pre-fetch data and a 1+1+X story fetching strategy to improve server-side processing and download times. The post also highlights how non-critical services were enqueued to run after initial story display, further reducing perceived latency.

Nov 2015Open networking advances with Wedge and FBOSSThis post details Meta's significant advancements in open networking with the introduction and deployment of Wedge, an open-source top-of-rack network switch, and FBOSS, its Linux-based control software. It highlights the disaggregation of networking hardware and software, enabling faster innovation and customization. The post elaborates on operationalizing Wedge and FBOSS for large-scale deployments, including custom tools for seamless software updates, ensuring nonstop forwarding with Warm Boot, and lessons learned from treating switches like servers. It also discusses crucial CPU protection mechanisms and future plans for scaling this open networking approach to higher speeds and larger aggregation switches.

Nov 2015Building the Presto communityThis post details the creation and open-sourcing of Presto, a distributed SQL engine for interactive analytic queries. It highlights Presto's architecture, its performance improvements over traditional map-reduce jobs, and its adoption by companies like Airbnb, Dropbox, and Netflix. The post emphasizes the growth of the Presto community since its open-sourcing in 2013, with an increase in contributors and the development of community-driven tools like Airpal. It also notes international adoption and ongoing efforts to further develop the project.

Oct 2015Performance instrumentation for Android appsThis post introduces Meta's approach to performance instrumentation for Android apps, addressing the limitations of manual instrumentation and built-in profiling tools. It details the development of a bytecode instrumentation system that automatically injects code to collect detailed performance telemetry from real-world usage. Key contributions include the ability to trace asynchronous operations across threads and the development of a lock-free ring buffer for efficient event collection, enhancing the understanding of performance bottlenecks in a rapidly evolving mobile environment.

Oct 2015Improving the Linux kernel with upstream contributionsThis post details Meta's "upstream-first" philosophy for Linux kernel development, emphasizing active participation in the upstream community to integrate their specialized kernel modifications for large-scale deployments. It highlights contributions in areas like blk-mq for storage performance, Btrfs filesystem stabilization, NV congestion control, cgroup writeback support, network virtualization (GUE, ILA), IPv6 performance improvements, and RAID 5/6 caching. The post showcases the team's commitment to improving the Linux kernel for both internal use and the broader community, enabling faster adoption of new technologies and reducing internal churn.

Aug 2015OpenBMC for server: Porting and supporting new features for “Yosemite”This post details the porting and feature additions to OpenBMC for the 'Yosemite' multi-node server platform. Key technical contributions include: removing the RMCP+ protocol for security, adding SSH and REST API support for management and debugging, implementing IPMI FRUID parsing for various Field Replaceable Units, developing a sensor monitoring daemon for analog sensors, updating software modules for multi-node support (including payload number in IPMI requests and IPMB framework), adding NCSI support to the Ethernet driver, enhancing the I2C driver for secondary mode and IPMB communication, adding IPMI SDR support with a caching daemon, implementing discrete signal monitoring for error conditions, and introducing new utilities (fruid-util, sol.sh, power-util, sensor-util) accessible via SSH.

Jul 2015Improving Facebook’s performance on Android with FlatBuffersThis post details the adoption and integration of FlatBuffers into Facebook's Android application to improve performance. It addresses the limitations of JSON parsing (speed, initialization, GC pressure) and explains how FlatBuffers' direct data access without deserialization, along with its mutation extensions and the resulting 'Flat Models' architecture, significantly reduced story load times from disk cache, transient memory allocations, and cold start times, while also reducing storage size.

Jul 2015Using ISC Kea DHCP in our data centersThis post details the migration of Facebook's data center DHCP infrastructure from ISC dhcpd to ISC Kea. It highlights the challenges with the previous system, including slow configuration propagation and frequent server restarts due to static configuration files. The adoption of Kea is driven by its modern design, extensibility via hook points, and suitability for a stateless deployment within containers. The new system fetches configuration dynamically from the inventory system, simplifying maintenance and accelerating hardware provisioning.

Jun 2015Recommending items to more than a billion peopleThis post details the development of a novel rotational hybrid approach for distributed collaborative filtering (CF) to scale to massive datasets (100 billion ratings, 1 billion users). It addresses limitations of standard distributed matrix factorization algorithms by partitioning items across workers and rotating them via worker-to-worker messaging, significantly reducing network traffic and eliminating skewed item degree issues. The approach is blended with the standard method for items with lower degrees and leverages JBLAS for matrix inversion in ALS. It also discusses combining SGD and ALS iterations for improved offline metrics.

May 2015Under the hood: Facebook’s cold storage systemThis post details the design and implementation of Facebook's cold storage system, a full-stack approach to storing and serving photos efficiently. Key technical contributions include: a new data center design with reduced power consumption and increased floor space; custom server hardware based on OCP specifications with controlled drive power cycles; modified Open Rack for higher density (2 PB per rack with 4TB drives) and lower power usage; software designed for durability without redundant power systems, using self-describing data; implementation of Reed-Solomon erasure coding for data redundancy with a 1.4 GB storage overhead for 1 GB of data; constant checksum validation and a background 'anti-entropy' process for detecting and repairing 'bit rot'; and a rebalancing service that treats new capacity as always available, improving durability and supporting migrations.

Mar 2015Even faster: Data at the speed of Presto ORCThis post details the development and integration of a new ORC reader for Presto, designed to significantly improve query performance. Key technical contributions include: implementing columnar reads to avoid row-to-column reorganization, enabling predicate pushdown by leveraging ORC's min/max statistics to skip irrelevant data segments, and introducing lazy reads to inspect only necessary columns for query filters. The new reader supports both ORC and DWRF formats and was benchmarked against previous Hive-based ORC readers, RCFile-binary, and Impala's Parquet reader, demonstrating substantial speedups (2-5.8x wall time, comparable CPU time) across various scales and compression schemes.

Mar 2015Introducing “Yosemite”: the first open source modular chassis for high-powered microserversIntroduced "Yosemite", an open-source modular chassis design for high-powered microservers, as a contribution to the Open Compute Project. Yosemite utilizes system-on-a-chip (SoC) processor cards targeting up to 65W TDP, with multiple memory channels, local SSD interface, and a management controller. It extends the OCP microserver interface with an additional PCI-E x16 connector and features a shared network connection for data and management traffic. The design supports up to 4 SoC cards per chassis, compatible with Open Rack, and aims to provide a cost-effective, flexible, and easy-to-service system for scale-out infrastructure.

Mar 2015Facebook Open Switching System (“FBOSS”) and Wedge in the openThis post announces the open-sourcing of FBOSS (Facebook Open Switching System) and the contribution of the Wedge switch specification to the Open Compute Project. FBOSS is a set of applications that run on a standard Linux OS to program and control network ASICs, enabling automation, configuration, monitoring, and troubleshooting. The Wedge switch is a disaggregated top-of-rack switch designed to be managed like a server. The post details the anatomy of a network switch, explaining the role of ASICs and the FBOSS agent in programming them. It also discusses the disaggregated model of network hardware and software, contrasting it with traditional monolithic network switches. The FBOSS agent's lean feature set, focused on L3 unicast routing and ECMP, is highlighted, along with its reliance on Broadcom's OpenNSL APIs. The post outlines future plans for FBOSS development, including adding control/routing protocol code and supporting other environments.

Mar 2015Introducing “OpenBMC”: an open software framework for next-generation system managementThis post introduces OpenBMC, an open software framework for next-generation system management, developed in-house due to the limitations of closed BMC software stacks. It details the motivation for building OpenBMC, its "under the hood" architecture as a customized Linux distribution built with Yocto Project, and its initial deployment on Facebook's Wedge switch. The post outlines the layered structure of OpenBMC (common, SoC, and board layers) and calls for community contributions to foster an ecosystem for BMC innovation.

Mar 2015Serving Facebook Multifeed: Efficiency, performance gains through redesignThis post details the redesign of Facebook's Multifeed system using a disaggregated hardware and software architecture. Previously, aggregators and leaf storage servers were co-located, leading to reliability issues, hardware scalability challenges, resource waste, and performance contention. The new design separates CPU-intensive aggregators onto specialized compute servers and memory-intensive leaf storage onto specialized memory servers. This disaggregation resulted in a 40% efficiency improvement in memory and CPU consumption, a 10% reduction in aggregator latency, independent scaling of components, and increased resilience to traffic spikes and component failures.

Feb 2015Introducing “6-pack”: the first open hardware modular switchIntroduced '6-pack', the first open hardware modular switch platform, designed to address the limitations of traditional networking technologies at scale. '6-pack' is a full mesh non-blocking two-stage switch composed of 12 independent switching elements, each capable of 1.28Tbps. It leverages 'Wedge' as a building block and features a split control configuration with hybrid SDN for management. The platform includes modular line cards and fabric cards, enabling the creation of switches of any radix using common building blocks. The design is intended to be contributed to the Open Compute Project.

Jan 2015Alibaba joins WebScaleSQLAlibaba joins the WebScaleSQL project, contributing bug fixes, performance improvements, enhanced large-scale parallelism, and micro-optimizations in compression for MySQL. The post also highlights ongoing WebScaleSQL development including replication and GTID server-side fixes, async client features, logical read-ahead, query throttling, improved server-level thread pooling, large InnoDB page support, and memory usage reduction.

Nov 2014Data @Scale – Boston RecapThis post details the Data @Scale – Boston conference, highlighting talks on real-time high-volume analytics processing with the Lambda architecture (Twitter), technical challenges and infrastructure enablers for large-scale product development (Facebook's Lookback videos), automated design tools for database projections (Vertica's DBD), scaling Redis and Memcached with composable tools for resilient distributed systems (Wayfair), big data business lessons for handling operations at scale (TripAdvisor), implementing spatial indexing on RocksDB for geo-spatial data (Facebook), Vertica Live Aggregate Projections for faster querying and incremental maintenance, building scalable caching systems with Mcrouter (Facebook, Instagram, Reddit), scaling Cassandra and MySQL (ConstantContact), comparing H-Store/VoltDB architecture with CEP and streaming architectures (VoltDB), using graph partitioning in distributed systems design (Facebook), and making enterprise software easy to install (Infinio).

Nov 2014Introducing data center fabric, the next-generation Facebook data center networkIntroduced the 'data center fabric', a next-generation network architecture for Meta's data centers. This fabric replaces the previous cluster-based design with a disaggregated approach using server pods and uniform high-performance connectivity via fabric switches and spine switches across four planes. Key technical details include the use of standard BGP4 routing with a distributed control, centralized override hybrid approach, layer3 and dual-stack (IPv4/IPv6) operation, heavy reliance on ECMP with flow-based hashing, and a multi-speed link strategy (40G inter-switch, 10G server). The design emphasizes modularity for rapid scaling of compute, network, and external connectivity, and allows for gradual scalability from an initial oversubscribed state to a non-oversubscribed network by adding spine switches. The physical infrastructure and cabling were optimized for rapid deployment.

Nov 2014Audience Insights query engine: In-memory integer store for social analyticsThis post details the design and implementation of the Audience Insights (AI) query engine, a distributed system for social analytics. Key technical contributions include: a hybrid integer store (in-memory and flash), fan-out distributed architecture with aggregation and leaf tiers, sharding by user, and a novel indexing strategy using bitsets for dense attributes and converted bitmaps for sparse attributes to accelerate filter operations. It also describes columnar-like data organization for improved cache efficiency, GPU acceleration for affinity computation, and tail latency reduction techniques for distributed query processing.

Sep 2014Lessons from Deploying MySQL GTID at ScaleThis post details the deployment of MySQL Global Transaction ID (GTID) at Facebook scale. It covers the benefits of GTID (failover, backup recovery, hierarchical replication), the challenges of deploying it to existing large-scale environments (non-dynamic gtid_mode), and the custom solutions developed in fb-mysql to enable a high-availability deployment strategy. Specific server-side fixes and automation changes are highlighted, including improvements to binary log index file format for faster initialization, a new transaction table for crash-safe GTID storage, and enabling multi-threaded secondary replication with relay_log_recovery=1 when GTID is enabled. Preparation steps like updating automation, auditing GTID-incompatible statements, and developing a skip statement script are also discussed.

Jun 2014Introducing “Wedge” and “FBOSS,” the next steps toward a disaggregated networkIntroduced "Wedge," a disaggregated top-of-rack network switch, and "FBOSS," a Linux-based operating system for it. "Wedge" uses a modular microserver architecture, allowing for interchangeable processors (Intel, AMD, ARM) and integration into existing fleet management systems. "FBOSS" provides a Thrift-based abstraction layer for switch ASIC APIs, enabling network devices to be treated as services and facilitating hybrid distributed/centralized control for improved utilization and faster recovery. It also integrates with existing environmental monitoring tools for performance insights.

Jun 2014Open-sourcing Haxl, a library for HaskellOpen-sourced Haxl, a Haskell library that simplifies access to remote data sources by automatically batching requests, fetching data concurrently from multiple sources, and caching previous requests. This addresses the expressiveness vs. performance trade-off in data-fetching code by leveraging Haskell's Applicative Functors for implicit concurrency and its caching mechanism for soundness and modularity.

Jun 2014HydraBase – The evolution of HBase@FacebookThis post introduces HydraBase, a novel evolution of HBase designed to significantly improve reliability and availability. HydraBase hosts each region on a set of region servers forming a quorum, leveraging the RAFT consensus protocol for consistency. This architecture allows for faster failover times by eliminating the need for log splitting and enables tolerance of multiple failures across racks or data centers. It effectively decouples logical and physical replication, aiming to increase availability from 99.99% to 99.999%.

Jun 2014Saving capacity with HDFS RAIDThis post details the implementation and lessons learned from deploying HDFS RAID (Erasure Codes) to reduce data replication factor and increase storage capacity. It introduces XOR and Reed-Solomon implementations, identifies the 'small file problem' and proposes 'directory RAID' as a solution. It also discusses challenges like data corruption, large directory handling, and directory changes, along with future work on native HDFS integration for RAID.

Apr 2014Large-scale graph partitioning with Apache GiraphThis post details the application of graph partitioning using Apache Giraph to solve data locality problems in sharded distributed systems at Facebook. It describes a heuristic algorithm, adapted from balanced label propagation, that iteratively swaps vertices to maximize local edges while maintaining partition balance. The algorithm uses a probabilistic approach for vertex relocation and a master computation to enforce balance constraints. It also highlights the benefit of initializing partitions based on geographical proximity and the ability to incrementally update partitions. The results show significant improvements in iteration times for PageRank and a 2x improvement in bandwidth utilization for a specific service.

Mar 2014WebScaleSQL: A collaboration to build upon the MySQL upstreamIntroduced WebScaleSQL, a collaborative effort to enhance MySQL for large-scale deployments. Key contributions include an automated testing framework (mtr), a new suite of stress tests, performance optimizations (buffer pool flushing, query optimizations, NUMA interleave policy), and operational features (super_read_only, sub-second client timeouts). Future work includes an asynchronous MySQL client, table/user/compression statistics, and Logical Read-Ahead mechanism.

Feb 2014An analysis of Facebook photo cachingThis post details the architecture of Facebook's multi-layered photo-serving stack, including browser caches, edge caches, origin cache, and the Haystack backend. It presents findings from an academic study on trace data, analyzing cache effectiveness, popularity distribution shifts across layers, the impact of cache size on hit ratios, and the performance improvements gained by implementing advanced caching algorithms like S4LRU over FIFO. The analysis quantifies the reduction in requests served by the Haystack backend due to caching and demonstrates the potential for further improvements through larger caches and smarter eviction policies.

Feb 2014An analysis of Facebook photo cachingThis post details the deep and distributed photo-serving stack at Facebook, analyzing the effectiveness of each cache layer, how photo popularity changes across these layers, and the impact of increasing cache sizes and advanced caching algorithms. It references an academic study published in SOSP.

Nov 2013Under the Hood: Building and open-sourcing RocksDBThis post introduces and open-sources RocksDB, an embeddable, persistent key-value store built on LevelDB. RocksDB is designed to scale on multi-core servers, efficiently utilize fast storage (especially flash), and support various workloads (IO-bound, in-memory, write-once). Key architectural features include a pluggable design for compression, compaction, and storage files, support for MVCC for read-only transactions, and a log-structured merge tree. Performance improvements over LevelDB are highlighted, particularly for IO-bound workloads on flash storage, due to reduced lock contention, better write amplification, and optimized compaction.

Nov 2013The Mature Optimization HandbookThis post introduces a handbook on mature optimization strategies, emphasizing the importance of measurement-guided decision-making for performance improvements in critical code paths (the 'critical 3%'). It advocates for a cost-benefit analysis to determine the value of optimization efforts, suggesting that performance work is often best suited for mature systems and can be deferred until after initial shipping, especially when resources are limited or post-release changes are difficult. The handbook also cautions against premature optimization and highlights the potential negative impacts on debugging and maintenance.

Oct 2013Under the Hood: Building posts searchThis post details the engineering challenges and solutions for building a search index capable of handling trillions of posts. Key contributions include: 1. Data Collection: Harvesting data from a production MySQL database with 70 different data types, carefully monitoring load and bypassing caches during initial collection. 2. Index Building: Storing harvested data in HBase and using Hadoop map-reduce jobs to create a search index, separating document data for ranking and the inverted index for search queries. 3. Index Updating: Subscribing to MySQL changes via Wormhole for real-time index updates, hitting caches during updates. 4. Serving the Index: Storing the large index primarily on solid-state flash memory, with frequently accessed data structures in RAM to maintain performance. 5. Ranking Results: Implementing query rewriting and dynamic result scoring using over a hundred ranking features combined with a ranking model to surface relevant content.

Oct 2013Under the hood: MySQL Pool Scanner (MPS)This post introduces the MySQL Pool Scanner (MPS), a state machine system written in Python for automating the management of Meta's large-scale MySQL database clusters. MPS handles routine DBA tasks, bulk maintenance, and high availability by automating instance copying/placement and primary instance promotion. It manages instance states (Production, Spare, etc.) and uses a 'problem' system to drive automated resolution of failures (e.g., broken secondaries, low disk space) and site-wide maintenance operations (e.g., kernel upgrades, data set replication to new data centers).

Oct 2013Flashcache at Facebook: From 2010 to 2013 and beyondThis post details the evolution and optimization of Flashcache, a Linux kernel device mapper target for L2 caching, at Facebook between 2010 and 2013. It describes the initial motivation to address cost and performance trade-offs between SAS/SATA and flash storage, and the decision to implement a software-layer solution. The post then dives into performance analysis and optimization efforts, identifying three major areas: read-write distribution (moving from linear mapping to random hashing and adjusting associativity sizes), cache eviction (evaluating LRU with mid-point insertion using blktrace data and settling on LRU-2Q at the 75th percentile), and write efficiency (implementing a unified dirty data eviction method that treats all pages equally and prioritizes oldest entries in the LRU). The authors also discuss future work on metadata structures, multi-TB cache devices, and fine-grained locking. The post quantifies improvements such as increasing average hit rate from 60% to 80% and cutting disk operation rate nearly in half, with specific systems seeing 40% read I/O reduction and 75% write I/O reduction.

Aug 2013Scaling Apache Giraph to a trillion edgesThis post details the adaptation and scaling of Apache Giraph to handle trillion-edge graphs at Facebook. Key contributions include: modifying Giraph for flexible vertex/edge input from separate sources (GIRAPH-155, GIRAPH-639), developing HiveIO for faster Hive table access (up to 4x), implementing multithreading for graph loading, computation, and result storage (GIRAPH-374, GIRAPH-615) to achieve near-linear speedups, optimizing memory usage by serializing vertices/edges to byte arrays (GIRAPH-417, GIRAPH-435) and using primitive data types (GIRAPH-528) to reduce overhead by up to 10x, and implementing sharded aggregators using Netty to bypass Zookeeper bottlenecks and handle large-scale distributed state aggregation.

Jul 2013Windex: Automation for database provisioningThis post introduces Windex, an automated system for database provisioning at Facebook. Windex handles wiping data from hosts coming out of production, reinstalling the OS and MySQL, and configuring them for the spares pool. It integrates with MySQL Pool Scanner (MPS) to manage the lifecycle of database hosts, from detection of issues to re-imaging and final checks before rejoining the spares pool. The system includes retry logic and failure pattern detection to prevent infinite loops and ensure hosts needing review are flagged. It also incorporates validation checks at multiple stages to prevent damage to production systems.

Jun 2013TAO: The power of the graphThis post introduces TAO (The Associations and Objects), a distributed graph database system designed to efficiently serve Facebook's social graph. It addresses the limitations of using Memcache and MySQL directly for this workload by providing a unified API for objects (nodes) and associations (edges). TAO implements a multi-tiered caching architecture (followers and leaders) with RAM and Flash memory, partitions data into shards stored in MySQL, and optimizes for creation-time locality. The API simplifies data access for product engineers, reducing complexity and improving the development cycle.

Jun 2013Wormhole pub/sub system: Moving data through space and timeThis post introduces Wormhole, a reliable publish-subscribe system designed to propagate changes across Facebook's infrastructure, both within and across data centers. Key properties include data partitioning (per shard streams), rewind-in-time capability for failure recovery, reliable in-order delivery (at-least-once), atomicity with database operations, low-latency streaming (milliseconds above network latency), and efficiency through multiplexing (caravans), sampling, and server-side filtering. Wormhole processes over 1 trillion messages daily, improving cache consistency, enabling real-time data warehousing, reducing UDB CPU/IO utilization by 40%/60%, and decreasing latency from a day to seconds.

Apr 2013Scaling memcache at FacebookThis post details the early adoption and scaling of memcache at Facebook, starting in 2005. It highlights memcache's critical role in managing database load and maintaining fast page load times for a rapidly growing user base. The post identifies key infrastructure challenges related to scaling social graphs, handling fan-out queries, and adapting to new product development. It also references a technical paper published in the 10th Usenix NSDI 2013 proceedings for deeper technical details.

Apr 2013LinkBench: A database benchmark for the social graphIntroduced LinkBench, a new database benchmark designed to simulate the transactional workload of social graph data. LinkBench replicates the data model, graph structure (power-law out-degree distribution), and request mix (read-dominated, heavy on edge operations and reads, particularly edge range scans) of Facebook's MySQL social graph workload. It includes a Java driver for generating the graph and operations, and supports adapters for evaluating different database systems. Benchmark results for MySQL 5.1.53 are provided, showing sustained throughput and latency characteristics under load.

Mar 2013Under the Hood: Indexing and ranking in Graph SearchThis post details the extension of the Unicorn inverted index framework to support search ranking capabilities for Graph Search. Key contributions include: maintaining different entity types in separate Unicorn verticals coordinated by a top aggregator, extending retrieval operations with weak 'and' and strong 'or' semantics, implementing query rewriting to translate user intent into structured Unicorn queries, adding scoring mechanisms to assign numeric scores to entities, introducing a forward index for additional entity metadata, implementing result set scoring to improve the diversity of results, developing a blending algorithm for combining results from different verticals, and extending support for nested queries to handle multi-vertical searches and data joining. A framework for A/B testing ranking experiments is also described.

Mar 2013Under the Hood: Building out the infrastructure for Graph SearchThis post details the technical challenges and solutions involved in building the infrastructure for Facebook's Graph Search. It describes the unification of multiple legacy search systems (PPS, Typeahead) into a single, scalable inverted-index system called Unicorn. Unicorn was extended to support graph traversal queries, enabling searches based on relationships between entities (nodes and edges) in addition to metadata. The post explains the index structure, posting lists, static rank for ordering entities, and the use of map-reduce for indexing and live update pipelines for real-time data freshness. It highlights the decision to use Unicorn for its scalability and efficiency in handling the vast and dynamic Facebook graph.

Mar 2013McDipper: A key-value cache for Flash storageIntroduced McDipper, a Memcache protocol-compatible key-value cache server optimized for flash storage. McDipper offers improved capacity and performance over RAM-based Memcached for large working sets with moderate to low request rates. Key features include configurable replacement policies (FIFO/LRU), bloom filters, compression, encryption, and arenas. Addressed Memcached race conditions with a delete hold-off mechanism and implemented a read-modify-write primitive for atomic operations. McDipper has been successfully deployed in production for photo infrastructure, achieving significant server reduction and sub-millisecond latencies.

Jan 2013Fighting spam with pure functionsThis post introduces FXL (Feature eXtraction Language), a domain-specific language optimized for spam fighting at Facebook. FXL leverages pure functions to enable automatic batching of data fetches and memoization of expressions, leading to a significant performance improvement (estimated factor of twenty speedup) compared to naive execution models. The language is a subset of Standard ML with customized syntax, designed for efficient data retrieval across the graph.

Jan 2013Under the Hood: Building Graph Search BetaThis post details the engineering challenges and solutions behind building Facebook's Graph Search beta. It describes the evolution from basic search systems to a comprehensive social graph search, the decision to build a single, scalable system rather than smaller targeted ones, and the development of a prototype parser. It highlights the infrastructure challenges, including scaling to over a billion users and consolidating three existing search systems into Unicorn. The post also discusses the critical privacy considerations and real-time checking required for complex queries. The future roadmap includes indexing posts and comments, and expanding to mobile and other languages.

Nov 2012Under the Hood: Scheduling MapReduce jobs more efficiently with CoronaThis post introduces Corona, a new scheduling framework developed to address limitations in Hadoop MapReduce's scheduling, job coordination, and resource management. Corona separates cluster resource management from job coordination by introducing a cluster manager and dedicated job trackers per job. It shifts from a pull-based to a push-based scheduling model, minimizing latency for small jobs. Corona also implements fair-share scheduling with improved fairness guarantees and supports non-disruptive upgrades. The deployment was phased across 500 nodes, then non-critical workloads, and finally all MapReduce jobs. Benefits include improved scalability, cluster utilization (reaching >95%), reduced slot refill times (17% improvement), and halved job latency for a test job. Future work includes resource-based scheduling and integrating other application types.

Oct 2012Interning at Facebook: Building the new mobile advertising interfaceThis post details the implementation of the first mobile advertising interface for Facebook within the Pages Manager for iOS application. It covers the client-side UI development using BoltJS, API endpoint design for scalability, and internationalization considerations for currencies and budget scales. It also highlights the collaborative effort with platform, internationalization, and payments teams, and the integration of a new page insights experience designed by another intern.

Oct 2012Under the Hood: Building the App Center recommendation engineIntroduced the App Center recommendation engine, detailing its aggregator-leaf architecture for handling large-scale data and traffic. Described scaling strategies through sharding and replication. Defined a quality metric based on user ratings and daily active users. Outlined algorithmic components: candidate selection (demographics, social, past interactions), scoring (explicit, dynamic, latent features), and real-time updates for user feedback, dynamic features, and latent features derived from low-rank matrix approximation.

Sep 2012Monitoring cache with ClaspinIntroduced Claspin, a monitoring tool that uses heatmaps to visualize the health of Facebook's cache systems (Memcache and TAO). Claspin encodes tribal knowledge about cache metrics (latency, request rate, error rate, timeouts, retransmits) into a visual format, allowing engineers to quickly identify problematic hosts and racks. The tool processes server-side stats and renders heatmaps in JavaScript using SVG, with hosts colored based on their 'hottest' statistic and predefined thresholds. Enhancements include support for additional backends, custom palettes, and revision-controlled configuration files, leading to increased adoption and improved incident prevention.

Aug 2012Ship early and ship twice as oftenThis post details the doubling of facebook.com's release frequency to twice daily, driven by the hiring of a new release engineer and the expansion of the release engineering team. This change aims to maintain the rapid release process established for 100 engineers as the company scales to 1000 engineers, ensuring engineers worldwide can ship code as quickly as their West Coast counterparts.

Jun 2012Under the Hood: Hadoop Distributed Filesystem reliability with Namenode and AvatarnodeThis post introduces Avatarnode, a highly available solution for the HDFS Namenode developed at Facebook. It addresses the Namenode's single point of failure (SPOF) by implementing a two-node setup with hot failover managed via ZooKeeper. Avatarnode allows Datanodes to send block reports to both Primary and Standby Namenodes, and modified clients check ZooKeeper for the current master, enabling writes to complete even during failover. This contribution was open-sourced to the Hadoop community.

May 2012Back-end build systems and toolchain: Adam SimpkinsThis post introduces fbconfig as a back-end build system that simplifies project creation and code sharing for developers by abstracting compilation and build dependency details. It also highlights ongoing work on optimizing the HTTP stack and load balancing infrastructure, including the addition of SPDY support, to improve network efficiency and performance.

Mar 2012Under the Hood: Improving Facebook PhotosThis post details the optimization of sRGB color profiles for Facebook photos to reduce file size and improve loading performance. The author, through a 'Hackamonth' project, collaborated with an image scientist to create a smaller, 524-byte sRGB profile (TINYsRGB) by removing unnecessary data and reducing the tone response curve sampling points from 1024 to 26. This optimization allows the profile to be attached to even small thumbnails without significantly impacting page load times, improving the visual accuracy of photos on Facebook.

Mar 2012MySQL and Database Engineering: Mark CallaghanThis post details the challenges and ongoing work in scaling MySQL on Facebook's multi-core servers with fast storage. It highlights the need to increase evictions per second from 1,000 to 10,000, the adaptation of innoDB compression for high-write workloads and faster storage, and the incremental improvement of checksum algorithms (e.g., using CRC32) for on-the-fly server upgrades. It also emphasizes the importance of making systems easily debuggable and predictable for reliability.

Jan 2012Building Timeline: Scaling up to hold your life storyThis post details the engineering effort behind Facebook's Timeline feature, focusing on the challenges and solutions for scaling to handle years of user activity. Key contributions include a massive data denormalization process to optimize database requests, the development of a C++-based Timeline aggregator for efficient story generation and caching, and the use of proven technologies like MySQL/InnoDB, Multifeed, Thrift, and memcached. It also highlights the parallel development strategy and close collaboration between product and engineering teams.

Dec 2011Meet a Facebook Engineer: Haiping ZhaoThis post introduces Haiping Zhao, a senior software engineer at Facebook, who discusses his work on scalability problems and the development of a distributed SQL database. He highlights the challenges of managing massive data growth and the inefficiencies of traditional single-machine databases. Zhao's work aims to address these issues by creating a database that scales effectively and handles data with different access patterns efficiently. He also mentions his prior work on HipHop for PHP, focusing on its development phases from ideation to testing and deployment.

Dec 2011Meet a Facebook Fellow: Marek OlszewskiIntroduced Kendo, a software-only prototype system providing a deterministic multithreading programming model. Kendo enforces deterministic interleaving of synchronization operations via a novel distributed scheduling algorithm that uses performance counters to construct logical clocks for deterministic and load-balanced synchronization schedules. It incurs only a modest performance cost and simplifies debugging by enabling deterministic reproduction of bugs.

Sep 2011Software Design GlossaryThis post introduces and defines key software design concepts: Element (generic term for design components), Coupling (interdependence of elements), Cohesion (internal unity of an element), Succession (breaking down conceptual changes into safe, ordered steps, exemplified by data store migration), Socialization (community absorption of change), Latency (time to first feature), Throughput (features per unit time), and Variance (unpredictable variation in latency/throughput). It emphasizes responsive design principles to optimize both latency and throughput by managing coupling and maintaining design cleanliness.

Sep 2011Interning at Facebook: Who Goes Where When (and Why It Matters)Developed an algorithm to predict business operating hours based on user check-in data. This algorithm involves making initial predictions based on an openness threshold, filling short gaps in predicted closed hours, and aggregating predictions across days. Implemented a user interface for suggesting business hours and integrated it into the Places Editor. Developed a 'check-in similarity score' to validate user-suggested hours. Added a 'Status' field to place pages to indicate current open/closed status and integrated this information into city pages.

Jul 2011Moving an Elephant: Large Scale Hadoop Data Migration at FacebookThis post details the successful migration of dozens of petabytes of Hadoop data between data centers. It describes the challenges of scaling Hadoop clusters beyond physical limitations and the decision to implement a replication-based migration strategy over a physical move to minimize downtime. The post outlines the two-step process: a bulk copy using DistCp with custom modifications for Facebook's dataset, followed by a new in-house replication system that polled a custom Hive audit log to capture and propagate file changes. It highlights the development of a multi-threaded replication system to handle millions of files and Hive objects, and the switchover process involving DNS changes and careful management of job submission gates. Key lessons learned include the importance of a fast replication system for disaster recovery and minimizing downtime.

May 2011How Project Triforce Prepared our Software Stack for PrinevilleThis post details the engineering effort behind Project Triforce, which simulated a third data center region to prepare Facebook's software stack for the new Prineville, Oregon facility. It highlights challenges in testing a complex, multi-region infrastructure, software complexity across hundreds of back-end services, new hardware configurations (Flashcache with MySQL), and the need to uncover implicit assumptions about regionality. The solution involved reconfiguring an active production cluster in Virginia to mimic a third region, testing with production traffic and simulated databases, and developing an in-house automation suite called Kobold for rapid cluster provisioning and deployment. This enabled the successful launch of the Prineville data center within a tight schedule, serving production traffic with minimal user impact.

Mar 2011Hybrid Incremental MySQL BackupsThis post details the transition from mysqldump (logical backup) to a custom physical backup model using XtraBackup for MySQL databases. Key enhancements include: implementing table-level restore via a PHP script, optimizing full and incremental backups by increasing IO size to 8MB (doubling incremental backup speed and making full backups 40% faster), and developing a "hybrid incremental" approach that uses a page tracker to reduce IO calls by 20-30% compared to standard incremental backups. The net impact shows significant improvements in backup speed and comparable space requirements to compressed logical backups.

Dec 2010Join Optimization in Apache HiveThis post details optimizations to join operations in Apache Hive. It introduces the use of Hadoop's Distributed Cache to store serialized in-memory hash tables for small join tables, solving scaling issues with traditional map joins. It also describes a mechanism to dynamically convert common joins into map joins based on execution-time file size, improving user experience by removing the need for manual hints. Performance improvements of 12-26x for optimized map joins and 57%-163% for converted common joins are reported. Tracking mechanisms for join conversion and CPU time saved are also introduced.

Dec 2010Visualizing FriendshipsThis post details the process of visualizing the social graph to understand the locality of friendships. It describes sampling data from Apache Hive, combining it with user city data, and then merging with city longitude and latitude. The author explores different visualization techniques in R, including plotting points and lines, and ultimately uses weighted lines with semi-transparency and great circle arcs to represent friendships between cities, revealing a detailed map of the world and highlighting the geographical distribution of human relationships.

Dec 2010Looking at the code behind our three uses of Apache HadoopThis post details Meta's specific enhancements and open-sourcing of Apache Hadoop 0.20 for three distinct use cases: data warehousing (scaling NameNode to 30 PB, 2 PB/day scans), real-time applications (low latency, HBase integration, click log streaming), and MySQL database backups. It highlights the AvatarNode for HDFS High Availability (contributed as HDFS-976), HDFS-RAID for storage reduction (contributed to MapReduce), and reliable HDFS file appends for HBase (contributed to 0.20-append branch). Future work includes cross-data center Hadoop (HighTide, HDFS-1432), slotless MapReduce, and JobTracker downtime-free updates.

Dec 2010The Full Stack, Part IThis post introduces the concept of a 'full-stack programmer' and emphasizes the importance of understanding system behavior across different layers for performance and optimization. It details how SQL query performance can vary based on database storage engines (InnoDB vs. MyISAM) and indexing. It also outlines key factors for visualizing system data flow, including working data size, average request size, request rate, mutation rate, consistency, locality, computation, latency, and contention. Finally, it provides a simplified model of computation, categorizing I/O operations by speed and capacity (registers/CPU cache, main memory, SSD, magnetic disk, network) and highlighting the significant performance impact of the memory/disk speed gap.

Jul 2010Scaling Facebook to 500 Million Users and BeyondThis post outlines foundational principles for scaling large systems, emphasizing horizontal scaling across machines, the importance of moving fast and making incremental changes, measuring all aspects of system performance, organizing into small independent teams, and ensuring control and responsibility are aligned. It highlights the challenges and strategies involved in handling exponential growth, such as dealing with 500 million users, 100 billion hits per day, and massive data volumes, and discusses how these principles have been applied to Facebook's growth.

May 2010The Life of a Typeahead QueryThis post details the architecture and implementation of Facebook's interactive typeahead search system, designed for 400 million users. It describes the front-end constraints (performance, UI richness), the backend architecture involving an aggregator service and specialized leaf services (global, graph), the use of Thrift for inter-service communication, and caching with memcached. It also highlights the importance of performance, recall, and relevance trade-offs, and the use of A/B testing and usability studies to refine the user experience.

Mar 2010Intro to Facebook SearchThis post introduces Facebook's in-house search service, highlighting its evolution to serve over 400 million users and 150 million queries daily. It details the complexities of user-centric search, emphasizing the importance of personal and social context in ranking results. Technical challenges discussed include ranking on the critical path, the absence of traditional query caching due to the [user, query] nature of requests, managing a large 'hot' index, and the need for live updates within seconds of profile changes. Future posts are promised to cover the details of the social context service, indexing, live updates, and data structures.

Feb 2010Internal Company DashboardThis post introduces Meta's internal dashboards framework, designed to improve the scalability of internal tool development and enhance employee productivity. It details a system that allows non-engineers to build dynamic custom pages in the browser without coding, and enables engineers to create reusable, isolated, and configurable widgets. The architecture is described with a high-level widget class example, highlighting how widgets fetch data and render HTML based on configuration. The framework also supports viewer-configurable dashboards, allowing users to customize their view of the same dashboard.

Dec 2009Real-World Web Application BenchmarkingThis post introduces and details Facebook's proprietary capacity testing process, 'Dyno,' designed to measure web application performance under real-world conditions. It contrasts Dyno's methodology with industry-standard synthetic benchmarks, highlighting the importance of application-specific architecture replication and organic traffic measurements. Dyno focuses on request duration as a function of throughput, identifying contention-induced latency and determining maximum request throughput. The post details system configuration comparisons between Intel's Harpertown/San Clemente and Nehalem/Tylersburg platforms, showing a 62% increase in RPS and a 44% improvement in performance per watt with the Nehalem platform. It emphasizes that memory access latency is a significant factor in overall throughput for Facebook's web applications and advocates for accurate workload profiling prior to evaluating new technologies.

Sep 2009Reducing markup sizeThis post details the optimization of the Universal Feedback Interface (UFI) markup size by reducing the redundant delivery of HTML for the 'Like' feature. The previous implementation sent two complete sets of markup for the Like/Unlike states, while the optimized version sends only the necessary markup for the current state and dynamically updates the sentence and link via an asynchronous call. This change reduced the average size of home.php by 20 kilobytes, an 8% reduction, without degrading user experience.

Aug 2009Designing the Facebook username land rushThis post details the system design and performance optimizations for Facebook's username land rush. Key contributions include the introduction of a separate memcache tier as the authoritative data source for username availability checks, eliminating database dependency for reads. It also describes the implementation of fault tolerance mechanisms for the memcache tier using Scribe logs for recovery and a secondary memcache lookup for increased reliability. Optimistic concurrency control was used to improve write performance, and Bloom Filters were considered but rejected. Load testing was performed at 10x the expected launch load, identifying bottlenecks in network configuration, database ID generation, and write traffic capacity. Contingency plans, including disabling less essential services, were prepared but ultimately not needed due to the robust design.

Jul 2009Hammering UsernamesThis post details the operational challenges and engineering solutions for launching the username feature to 200 million users simultaneously. It highlights the scaling of Memcached infrastructure with 1TB of in-memory cache dedicated to username availability checks and blocked names. It also describes the operational strategies employed, including launching at low site activity, assembling a 'War Room' with cross-functional teams, performing background load testing, and developing 'dark launching' and 'nuclear option' levers to dynamically adjust site functionality and manage extreme traffic loads. The successful launch without resorting to 'nuclear options' is attributed to these preparations.

Dec 2008Scaling memcached at FacebookThis post details modifications made to memcached and the Linux kernel to scale Facebook's memcached deployment to over 800 servers and 28 terabytes of memory. Key contributions include: implementing a per-thread shared connection buffer pool for TCP/UDP sockets to reclaim gigabytes of memory per server, switching to UDP for get operations with per-thread socket sharing to address Linux UDP performance issues caused by lock contention, introducing opportunistic polling of network interfaces to distribute network processing evenly across all cores and mitigate single-core saturation, moving stats collection to per-thread to eliminate global lock contention on multi-core machines, and modifying the network device transmit queue dequeue algorithm to batch dequeues and reduce lock contention, enabling scaling to 8 threads on an 8-core system. These changes resulted in a significant increase in UDP request handling capacity and a reduction in average latency.

Oct 2008Facebook’s Scribe technology now open sourceThis post introduces Scribe, a new open-source system for collecting and processing large volumes of server data. Key design decisions include a flexible directed graph topology where servers only know the next hop, disk spooling for reliability (accepting a small risk of data loss for scalability), and a simple two-string data model (category and message) to avoid complexity. Scribe was built using Thrift to accelerate development and support multiple languages. It handles tens of billions of messages daily across over 100 use cases.

Aug 2008Scaling OutThis post details the initial steps taken to scale Facebook's infrastructure by establishing a new datacenter in Virginia to reduce latency and improve disaster recovery. It describes the challenges and solutions for cache consistency in a multi-datacenter setup, involving modifications to MySQL's replication stream to include memcache keys. It also outlines the traffic routing strategy using Layer 7 load balancers and cookies to manage writes to primary databases and account for replication lag.

What this trail covers