Scalability and Large-Scale Systems Engineering

7/15/2026

Introduced Hierarchical Interest Representation, an upstream representation layer for Meta Ads deep funnel optimization. This system utilizes a transformer-based graph learning architecture with bias-aware attention and self-supervised cross-view distillation to learn multi-hierarchical interest representations. It blends real-world knowledge (from multimodal advertiser and product content processed through LLMs) with engagement signals to enrich sparse interactions and generalize to rare entities. The system outputs universal embeddings for ads entities and Bag-of-Meaning interest tokens, trained end-to-end on billions of interactions. Key technical components include an enriched engagement graph (typed, weighted, time-decayed, heterogeneous), a transformer-based hierarchical encoder with graph-structural biases, and attention kernels designed for large-scale graphs.

7/13/2026

This post details the modernization of the Meta Ads Service by adopting sched_ext, an open-source, BPF-based extensible kernel scheduler. It highlights how this custom scheduling policy, tailored to the Ads delivery workload, achieved a 28% reduction in p99 latency, 3.28 MW power savings, and a 1.1% increase in ads ranked. The post emphasizes the strategic shift from a short-term fix for kernel upgrade regressions to a continuous optimization platform, enabling rapid iteration of scheduling improvements through user-space BPF programs and contributing sched_ext as a shared industry asset.

5/26/2026

Introduced SilverTorch, a unified model-based retrieval system for recommendation engines, replacing a microservice-based architecture with an 'Index as Model' paradigm. This involves reimplementing all retrieval components (ANN search, eligibility filtering, scoring) as pure PyTorch modules within a single neural network. Key technical contributions include the Bloom index filter for efficient GPU-based filtering and fused Int8 ANN search for reduced memory usage and improved serving performance, enabling higher throughput and cost efficiency.

4/21/2026

This post details the modernization of Facebook Groups Search by implementing a hybrid retrieval architecture that combines lexical search (Unicorn inverted index) with semantic search (Faiss ANN search using dense vector representations from a 12-layer, 200-million-parameter model). It introduces an L2 ranking stage using a Multi-Task Multi-Label (MTML) architecture to jointly optimize for clicks, shares, and comments. Additionally, it describes the integration of automated offline evaluation using Llama 3 to grade search results, addressing the challenges of validating semantic search quality at scale. Future work includes applying LLMs in the ranking stage and developing adaptive retrieval strategies.

4/16/2026

Introduced a unified AI agent platform for Meta's Capacity Efficiency Program. This platform leverages encoded domain expertise and standardized tool interfaces to automate the finding and fixing of performance issues, both proactively (offense) and reactively (defense). The system has recovered hundreds of megawatts of power by compressing manual investigation time from hours to minutes and automating the creation of pull requests for resolutions. The architecture uses a two-layer approach: MCP Tools for standardized interfaces and Skills for encoding domain expertise, enabling both offense and defense use cases with shared infrastructure.

4/6/2026

Introduced a pre-compute engine with specialized AI agents to systematically read codebase files and produce concise context files, encoding tribal knowledge. This resulted in structured navigation guides for 100% of code modules, documented over 50 non-obvious patterns, and reduced AI agent tool calls by 40%. The system includes an orchestration layer for natural language routing and self-refreshing mechanisms to maintain context freshness.

3/31/2026

Introduced the Meta Adaptive Ranking Model to serve LLM-scale RecSys models for ads. Key innovations include inference-efficient model scaling via request-centric architecture and sharing, model/system co-design with hardware-aware architectures and selective FP8 quantization, and reimagined serving infrastructure enabling O(1T) parameter scaling with multi-card GPUs. This addresses the inference trilemma by balancing model complexity with low latency and cost efficiency, achieving sub-second latency and improving hardware utilization.

3/2/2026

This post details Meta's journey in optimizing FFmpeg for large-scale media processing. Key contributions include: 1. Collaborating with FFmpeg developers to upstream threaded, multi-lane transcoding and real-time quality metric computation features, enabling the deprecation of Meta's internal FFmpeg fork. 2. Implementing support for Meta's custom ASIC, MSVP, through standardized FFmpeg hardware acceleration APIs. 3. Driving significant refactoring in FFmpeg for improved parallelism in video encoding.

2/9/2026

This post introduces Backend Aggregation (BAG) as a key technology enabling Meta's gigawatt-scale AI clusters like Prometheus. BAG is a centralized Ethernet-based super spine network layer that interconnects multiple spine layer fabrics across data centers and regions, supporting petabit-range capacities. It connects regional networks to Meta's backbone, enabling mega AI clusters. The post details how BAG layers are distributed regionally and interconnected using planar or spread topologies. It explains how BAG connects to L2 fabrics built with Disaggregated Schedule Fabric (DSF) and Non-Scheduled Fabric (NSF), discussing oversubscription ratios. Technical details include the use of modular chassis with Jericho3 (J3) ASIC line cards, eBGP with link bandwidth attributes for routing, UCMP for load balancing, and MACsec for security. Resilience is addressed through failure domain analysis and strategies to mitigate blackholing. The post also highlights the use of deep buffer switches for BAG to handle long cable distances and support PFC.

11/21/2025

Introduced Zoomer, a comprehensive, automated debugging and optimization platform for AI training and inference workloads at Meta. Zoomer provides deep performance insights, enabling energy savings, workflow acceleration, and efficiency gains across Meta's AI infrastructure. It features an Infrastructure and Platform Layer for enterprise-grade scalability, an Analytics and Insights Engine for specialized analyzers (GPU, CPU, communication, memory), and a Visualization and User Interface Layer for intuitive data presentation. Zoomer supports various profiling triggers, comprehensive data capture (GPU metrics, execution traces, host-level data, application annotations, inference-specific data, communication analysis), and distributed analysis pipelines for straggler detection, bottleneck analysis, critical path analysis, anti-pattern detection, parallelism analysis, memory analysis, and load imbalance analysis. Specialized features include single-click QPS optimization for inference, request-level deep dives, real-time memory profiling, and LLM Zoomer for large-scale GenAI workloads.

11/11/2025

Introduced StyleX, a new styling system for Meta's large-scale applications. StyleX acts as a compiler that extracts styles at build time and generates static, collision-free atomic CSS, combining CSS-in-JS ergonomics with static CSS performance. It addresses historical challenges with CSS at Meta, such as collisions, dependency management, and large bundle sizes. Key technical contributions include the Babel plugin compiler, atomic CSS generation, stylex.create() for defining static styles, stylex.props() for merging and deduplicating styles, and APIs for shareable values (defineVars, defineConsts) and theming. The system enforces constraints to encourage predictability and enable composition, while preserving CSS features like media queries and pseudoclasses. It also supports dynamic styles via CSS variables and provides APIs for animations and view transitions.

11/4/2025

This post details the engineering challenges and solutions for scaling invisible watermarking at Meta. It describes the transition from GPU-based inference to a CPU-optimized solution, highlighting the identification and mitigation of bottlenecks related to data transfer, inference latency, and model loading. The post also discusses optimization considerations and trade-offs between latency, watermark detection accuracy, visual quality, and compression efficiency (BD-Rate), and introduces a novel frame-selection method to manage BD-Rate impact and a custom post-processing technique for visual quality evaluation.

10/20/2025

This post introduces Disaggregated Scheduled Fabric (DSF), Meta's next-generation network fabric technology designed for AI training networks. It details the challenges with traditional IP fabrics for AI workloads (elephant flows, low entropy, suboptimal utilization) and how DSF addresses them through a disaggregated architecture (interface nodes and fabric nodes), packet spraying, and a credit-based congestion control algorithm with Virtual Output Queuing (VOQ). The post also describes the scaling of DSF from single AI zones to dual-stage fabrics (L2 zones) and regional interconnectivity (L3 super-spine), highlighting the use of FBOSS and OCP-SAI. A key feature, Input Balanced Mode, is explained as a mechanism to prevent congestion during link failures by dynamically adjusting traffic based on reduced reachability.

10/16/2025

Introduced and open-sourced directory branching in Sapling, a source control system for Meta's monorepo. This new branching model addresses scalability and developer experience tradeoffs by allowing directory-level branching within a single repository. Key technical contributions include the `sl subtree copy`, `sl subtree graft`, and `sl subtree merge` commands, which leverage metadata to reconstruct relationships between directory branches and enable three-way merges. The implementation integrates with build systems like Buck2 via config modifiers and addresses code search challenges. The post details use cases such as managing development and production versions, facilitating experimental development, and unblocking Git migrations.

10/14/2025

This post details Meta's use of AI, specifically NLP and LLMs (Llama 3.1), to improve Scope 3 emission estimates for IT hardware. It introduces a methodology for identifying similar components using TF-IDF and Cosine similarity to leverage existing Product Carbon Footprints (PCFs). It also describes using LLMs to extract data from heterogeneous sources for parameterized modeling and to apply generative AI as a categorization algorithm for creating a new, standard taxonomy of IT hardware emissions. The post highlights the benefits of these AI approaches in enhancing data quality, consistency, and traceability for carbon footprint calculations, ultimately aiming to drive sustainable practices in the IT hardware industry.

10/14/2025

This post details advancements in Meta's data center networking for AI training clusters, including the evolution of Disaggregated Scheduled Fabric (DSF) to a dual-stage architecture supporting up to 18,432 XPUs, and the introduction of a new Non-Scheduled Fabric (NSF) architecture based on shallow-buffer Ethernet switches for large AI clusters. It also announces new OCP switch platforms (Minipack3N based on NVIDIA Spectrum-4), updates to FBOSS and SAI to support these fabrics, and new optical solutions (2x400G FR4 LITE, 400G DR4, 2x400G DR4). The post highlights Meta's leadership in the new Ethernet for Scale-Up Networking (ESUN) initiative within OCP, aiming to advance Ethernet for AI system interconnects.

10/6/2025

Introduced OpenZL, a new open-source format-aware compression framework. OpenZL applies a configurable sequence of reversible transforms to structured data to reveal patterns before compression, aiming for higher compression ratios and speeds than generic compressors. It features an offline trainer to generate compression configurations based on data shape descriptions (e.g., using SDDL) and a universal decoder that can decompress files compressed with various configurations. The system supports runtime adaptation through control points in compression plans to handle data variations and includes a Managed Compression system for continuous retraining and rollout of new configurations.

10/1/2025

This post details Meta's adoption and implementation of Android's Baseline Profiles to enhance the performance of its Android applications. It explains the challenges of ART's class loading and method compilation, Meta's specific performance hurdles with large class sets and frequent updates, and how Baseline Profiles, generated from user data and tuned by Meta, overcome these by enabling install-time optimizations. The post highlights improvements in startup performance and user journeys, contributing to a significant performance uplift across Meta's apps.

9/30/2025

Introduces the Automated Compliance Hardening (ACH) tool, an LLM-based system for software testing that automates compliance adherence. ACH addresses the challenges of scaling mutation testing by generating fewer, more realistic, and highly specific mutants targeted at particular fault classes. It also features an LLM-based Equivalence Detector agent to identify and filter out equivalent mutants, and automatically generates unit tests that kill the mutants. The tool has been trialed on platforms like Facebook, Instagram, and WhatsApp for privacy testing use cases.

9/29/2025

This post details Meta's infrastructure evolution, highlighting the shift from scaling traditional software and physical infrastructure to accommodating AI workloads. It emphasizes the impact of GPUs and AI accelerators on recommendation systems and the exponential increase in compute demands driven by Large Language Models (LLMs), requiring specialized AI clusters and holistic infrastructure planning.

9/26/2025

This post from @Scale:Networking 2025 recap highlights the increasing importance of networking infrastructure in enabling large-scale AI advances. It details Meta's significant investments in AI infrastructure, including gigawatt-scale clusters and global fiber cable systems, and discusses the rapid evolution of AI workloads (e.g., mixture-of-experts, reasoning models) and their specific network requirements. The post emphasizes the network's role as an abstraction layer, the need for co-designing networks with the AI stack, and the paramount importance of reliability and continuous innovation in this rapidly evolving field.

8/26/2025

This post details the integration of the Kotlin incremental compiler into Meta's Buck2 build system. Key technical contributions include: integrating with the Kotlin Build Tools API (KEEP), working around classpath conflicts by un-shading and then fully migrating to shaded Kotlin compilers, configuring Buck2's incremental actions to preserve previous build outputs, making the compiler cache relocatable for distributed builds, implementing source and dependency change tracking using Buck2's incremental actions and classpath snapshots, and ensuring compiler plugins and annotation processors (KSP2) work correctly with the incremental compilation process, including handling incomplete results and multiple compilation rounds.

8/11/2025

Introduces the Federation Platform and Privacy Waves program as a scalable system for managing compliance-related tasks, including privacy, security, and accessibility. Details the technical configuration of workstreams within the Federation Platform, including scraping flags, ownership resolution, grouping, actioning (task/diff generation), task content and distribution, and resolution logic. Outlines the end-to-end federation process, from strategy and task configuration to distribution, accountability, and reporting, highlighting the use of AI for support and summarization, and sophisticated matching algorithms for task assignment.

7/22/2025

This post details Meta's methodologies for detecting and mitigating Silent Data Corruptions (SDCs) in its AI hardware infrastructure. It categorizes hardware faults into static, transient, and silent errors, highlighting the unique challenges posed by SDCs in AI training (NaN propagation, corrupted gradient variance) and inference workloads. The post introduces and explains three novel SDC detection mechanisms: Fleetscanner, Ripple, and Hardware Sentinel, and discusses their impact on ensuring the reliability of large-scale AI systems.

5/21/2025

This post details Instagram's journey to scaling its recommendation system to over 1000 ML models. Key contributions include the implementation of a model registry to manage model metadata, importance, and business function; the development of automated model launch tooling that reduces launch time from days to hours; and the operationalization of 'model stability' as a metric to measure prediction accuracy and define SLOs. The post also describes the ranking funnel architecture (sourcing, ESR, LSR) and the challenges of managing a large fleet of models due to constant experimentation.

5/8/2025

This post details the integration of NVIDIA cuVS into Faiss v1.10, a Meta-developed open-source library for efficient vector search. The collaboration with NVIDIA has resulted in significant performance improvements for both inverted file (IVF) and graph-based indexing. Benchmarks show up to 4.7x faster IVF index build times and 8.1x reduced search latency for IVF PQ. For graph indexing, CUDA ANN Graph (CAGRA) achieves up to 12.3x faster build times and 4.7x reduced search latency compared to CPU-based HNSW. This integration enhances Faiss's capabilities for large-scale similarity search on GPUs.

3/31/2025

This post discusses the evolution and future of Mobile GraphQL at Meta, a framework used for data fetching in mobile applications. It highlights how GraphQL contributes to building better user experiences and making developers' lives easier through innovative APIs. The post also touches upon the insights and unexpected challenges encountered by the Mobile GraphQL Platform Team when building and scaling such a platform.

3/4/2025

This post introduces the evaluation and adoption of Quad-Level Cell (QLC) SSDs in Meta's data centers as a solution for increasing data density and power efficiency. It details the technical trade-offs between HDDs, TLC SSDs, and QLC SSDs, highlighting QLC's advantages in density, power efficiency, and cost for read-bandwidth-intensive workloads. The post also discusses hardware considerations, such as form factors (U.2 vs. E1.S/E3), and software adaptations required for high-density QLC servers, including I/O scheduling and rate control to manage the significant delta between read and write throughput.

2/3/2025

This post details Meta's approach to handling leap seconds within Precision Time Protocol (PTP) environments, which require nanosecond-level precision. It contrasts PTP's needs with traditional NTP leap second smearing methods (quadratic and linear) and introduces Meta's 'self-smearing' algorithm within the fbclock library. The post also highlights the trade-offs of using UTC over TAI for timekeeping in these high-precision systems and advocates for a future without leap seconds to simplify infrastructure and enhance timekeeping precision.

1/21/2025

Introduced Strobelight, a profiling orchestrator that integrates multiple profilers (many eBPF-based) to collect detailed performance metrics (CPU usage, memory allocations) from production hosts. Strobelight enables engineers to identify performance bottlenecks, optimize code, and improve resource utilization, leading to significant capacity savings. It supports ad-hoc profilers via bpftrace scripts and provides default continuous profiling with dynamic sampling and sample weighting for aggregation. Key components contributing to efficiency include the LBR profiler for FDO and the event profiler for stack trace analysis. Stack Schemas enhance data by adding tags to call stacks for better visualization and analysis.

1/10/2025

This post introduces Meta's 'ILA Evo' effort, a significant initiative to reimagine fiber optic in-line amplifier (ILA) sites. It details the historical context of ILA development and outlines the new requirements for ILA Evo, including faster deployment, reduced material usage, improved power efficiency (PUE < 1.5), and the adoption of lightweight building systems (FRP/GRP panels), low-concrete foundations (helical screw piles), manufactured owner-furnished equipment (MOFE) for ISP installation, advanced cooling technologies, and modern backup power alternatives (H2 fuel cells, molten salt batteries). The post also highlights the macro benefits of this disaggregated supply chain for improved commercial and risk management.

12/19/2024

Introduced Glean, an open-source code indexing system for collecting and querying facts about source code. Detailed Glean's design rationale, emphasizing language and use-case generality, and its use of RocksDB for storage and Angle for querying. Explained incremental indexing to achieve O(changes) indexing cost and discussed its application in code navigation, symbol search, and speeding up IDEs.

12/18/2024

This post details how the Threads iOS team approaches and maintains app performance during rapid growth. It introduces key performance metrics like %FIRE (frustrating image-render experience), TTNC (time-to-network content), and cPSR (creation-publish success rate). It also discusses the development of the SLATE debugger for consistent navigation latency measurement and a case study on validating GraphQL adoption for network requests, highlighting the importance of performance analysis in scaling new features and platforms.

12/10/2024

This post details the technical evolution of Facebook's video delivery system, focusing on the two-year effort to unify Watch and Reels. It describes the three core components: Ranking (RecSys), Server (WWW), and Mobile (FB4A/FBiOS). Key technical challenges and solutions are highlighted, including the unification of client and server data models and architectures, the migration of users to a unified Reels surface, and the unification of ranking layers to handle mixed inventory of short-form and long-form video. The post also outlines the five core principles guiding video delivery and the lifecycle of a video feed network request, from mobile client initiation to server processing, emphasizing prefetching, head loads, and tail loads with caching and user signal integration.

12/2/2024

Introduced Andromeda, a next-generation personalized ads retrieval engine that significantly enhances Meta's ad recommendation system. Andromeda leverages custom-designed deep neural networks optimized for NVIDIA Grace Hopper Superchip and MTIA hardware, achieving +6% recall improvement and +8% ads quality improvement. It employs hierarchical indexing to manage exponential ad creative growth from Advantage+ automation and generative AI, and introduces model elasticity for agile resource allocation, boosting model inference efficiency by 10x. The system achieves over 100x improvement in feature extraction latency and throughput compared to previous CPU-based components, and enhances end-to-end model inference QPS by over 3x.

11/19/2024

This post details the transition of Meta's ad recommendation system from traditional DLRMs with human-engineered features to a new paradigm based on sequence learning. Key technical contributions include the introduction of Event-Based Features (EBFs) as a standardized input for sequence learning models, the development of event models to synthesize event embeddings, and the use of sequence models with attention mechanisms (specifically multi-headed attention pooling to reduce complexity from O(N*N) to O(M*N)) to process these embeddings. The post also highlights scaling challenges and solutions, such as native PyTorch support for jagged tensors, kernel-level GPU optimizations, and a Jagged Flash Attention module. It further discusses scaling with longer and richer event sequences using multi-precision quantization, value-based sampling, and customized vector quantization for multimodal content.

10/15/2024

Introduced Disaggregated Scheduled Fabric (DSF) for next-generation AI clusters, a disaggregated and open network fabric. Contributed two new disaggregated network fabrics (Arista 7700R4 series, Minipack3/Cisco 8501) and a new NIC (FBNIC) to OCP. DSF leverages VoQ-based switched systems, OCP-SAI, and FBOSS for proactive congestion avoidance and high-bandwidth AI clusters. FBNIC is a multi-host foundational NIC with datapath isolation and HW offloads, with its driver upstreamed and module contributed to OCP. Enhanced optics to 2x400G FR4 for 400G/800G interconnection.

8/5/2024

This post details Meta's development of specialized data center networks for distributed AI training, focusing on the implementation and evolution of RoCEv2 for inter-GPU communication. It covers the network topology (AI Zones, aggregator switches), routing challenges and solutions (ECMP, path pinning, Enhanced ECMP with QP scaling), and congestion control strategies (PFC, receiver-driven traffic admission) to support large-scale AI workloads like LLMs.

8/5/2024

Introduced DCPerf, an open-source benchmark suite for hyperscale compute applications. DCPerf represents diverse workload categories in data center cloud deployments, designed by referencing large applications within Meta's production server fleet. It uses techniques to ensure benchmark representativeness, from low-level hardware microarchitecture features to application and library usage profiles. DCPerf has been enhanced for compatibility with x86 and ARM architectures, supports multi-tenancy, and has been used internally to improve compute server designs, aid in capacity planning, identify performance bugs, and jointly optimize platforms with hardware industry collaborators. It provides richer application software diversity and better coverage signals on platform performance compared to existing benchmarks like SPEC CPU.

7/10/2024

This post details optimizations to address tail utilization in Meta's ads inference services. Key contributions include tuning load balancing mechanisms with the 'power of two choices' using polling to avoid heavily loaded hosts, and implementing system-level changes. These system changes involved considering memory bandwidth as a resource during replica placement in Shard Manager and resolving expectation mismatches between ServiceRouter and Shard Manager regarding load balancing assumptions. These efforts resulted in a two-thirds reduction in timeout error rates, a 35% increase in work output for the same resources, and a halving of p99 latency.

7/10/2024

This post details Meta's systematic framework for building prediction robustness in its machine learning systems, specifically for advertising recommendations. It outlines the challenges of ML robustness (stochastic nature, frequent model refreshes, blurred reliability/performance lines, cumulative effects, complex interactions, amplification of small changes, and rapid innovation). The framework includes prevention guardrails, fundamental understanding, and intrinsic robustness applied across models, features, training data, calibration, and interpretability. Key technical contributions include Snapshot Validator for model snapshot quality, new ML techniques for intrinsic model robustness (pruning, generalization, quantization), robust feature monitoring systems, dedicated training-data-quality systems, high-precision alert systems for calibration, and the Hawkeye AI debugging toolkit for ML interpretability. These efforts aim to improve ML ranking performance, boost engineering productivity, and reduce the time to root-cause ML prediction issues.

6/11/2024

This post details the expansion of Meta's MobileConfig system to support mixed reality (MR) devices, including Meta Quest and Ray-Ban Meta smart glasses. It addresses configuration challenges in the MR ecosystem, such as extended development cycles and the need for shared configuration values across multiple apps. The post explains how MobileConfig was adapted by centralizing configuration requests through a single Android service, developing cross-platform libraries, and optimizing for low-powered devices and microcontrollers with limited connectivity and resources. Key technical adaptations include repurposing libraries for microcontrollers, optimizing memory footprint, and developing customized IPC APIs and AOSP Java/Native APIs for OS-level experimentation. The goal is to enable rapid innovation, experimentation, and remote control for future devices.

2/6/2024

Introduced DotSlash, a tool for simplified executable deployment that replaces large executables with lightweight descriptor files. DotSlash handles transparent fetching, decompression, and verification of platform-specific remote artifacts, reducing repository size and improving clone times. The post details the internal use cases at Meta, including managing first-party and third-party command-line tools within a monorepo and for core developer tools like Buck2 and Sapling. It also describes the DotSlash file format, its execution workflow, caching mechanisms, and the automated generation of DotSlash files via CI pipelines and GitHub Actions.

1/29/2024

This post details improvements to the ML/AI development workflow by addressing slow build times and inefficiencies in packaging and distributing executable files. Key contributions include: 1. Mitigating build non-determinism using Buck2 and Remote Execution to ensure consistent build outputs and enable warm revisions. 2. Optimizing dependency management by reducing dependency counts, removing unused code, and identifying used Python modules. 3. Implementing a Content Addressable Filesystem (CAF) for incremental packaging and distribution of Python executables, leveraging a CAS daemon with P2P distribution via Owl and Btrfs for efficient storage. This resulted in double-digit percentage reductions in overhead.

12/19/2023

This post details how Meta leveraged its existing, mature infrastructure, specifically ZippyDB and Async, to support the rapid launch and unprecedented growth of Threads. It highlights ZippyDB's multi-tenancy, common hardware pool, and resharding protocol for scaling keyspaces with zero downtime, and Async's serverless function platform for handling computationally expensive, non-blocking operations like social graph synchronization and notifications. The post emphasizes how these pre-existing, scalable systems enabled Threads to handle millions of users within days without requiring new infrastructure development specifically for the product.

10/24/2023

Introduced the Systematic Code and Asset Removal Framework (SCARF) with a subsystem for identifying and removing dead code. SCARF combines static and dynamic analysis (leveraging Glean for code dependency graphs and operational logs for API usage) across multiple programming languages (Java, Objective-C, JavaScript, Hack, Python). It operates at a symbol level for granular analysis and automatically generates code change requests via CodemodService. The system has evolved to analyze the complete augmented dependency graph, leading to a significant increase in dead code removal. It also incorporates application-specific semantics and domain-specific logging to fine-tune dependency analysis. SCARF has automatically deleted over 100 million lines of code.

10/23/2023

This post introduces Buck2, the successor to Meta's open-source build system Buck. Key technical details include: Buck2 is written in Rust for performance and memory control, with rules in Starlark. It supports remote execution to avoid downloading intermediate outputs, saving bandwidth and disk space. Buck2 defines 11 distinct file path types for enhanced type safety. It is language-agnostic, with language-specific logic defined in Starlark. Buck2 introduces BXL (Buck2 Extension Language) for inspecting and interacting with the build graph, enabling custom build actions and IDE project generation.

10/17/2023

Introduced the Systematic Code and Asset Removal Framework (SCARF), Meta's internal tool and framework for automating product and feature deprecation. SCARF guides engineers through the process of safely and efficiently removing code and data associated with deprecated products. It combines workflow management tools with automation to reduce engineer load, having removed petabytes of unused data and over 100 million lines of code. SCARF analyzes code and data dependencies, identifies internal and external dependencies, and creates deletion roadmaps to ensure a staggered, multi-step removal process. It leverages static and dynamic analysis metrics to determine which assets are in use and which are safe for automated deletion, facilitating a collaborative approach between engineers and automation.

9/7/2023

This post details the rapid development and launch of Meta's Threads app, highlighting the use of Python (Django) for the backend, Swift for iOS, and Jetpack Compose for Android. It emphasizes leveraging existing infrastructure and data models for speed, the importance of a startup-like environment for a small engineering team, and the use of internal monitoring tools like ODS and Scuba during launch. The post also discusses the future direction of Threads, focusing on decentralization and integration with ActivityPub.

8/9/2023

This post details the evolution of the Instagram Explore recommendations system, focusing on scaling and flexibility improvements. Key technical contributions include the adoption of a multi-stage ranking approach (retrieval, first-stage ranking, second-stage ranking, final reranking), the extensive use of Two Towers neural networks for both retrieval and first-stage ranking due to their cacheability, and the implementation of a multi-task multi-label (MTML) neural network for second-stage ranking. The post also highlights the use of user interaction history for candidate retrieval, pre-computation for heavy models during peak hours, and a value model (VM) for combining engagement probabilities into a final score. Finally, it touches upon final reranking for integrity and diversity.

2/7/2023

This post details the evolution of Meta's global basemap since its launch in late 2020. It highlights significant upgrades in data storage, the introduction of dark-mode and raster-first basemaps for faster rendering across devices, and the integration of open-source data from OpenStreetMap (Daylight) and the upcoming Overture Maps Foundation. The post also discusses improvements in cartographic design, including a toned-down color palette, deprioritization of roads to focus on community spaces, and the introduction of detailed pavement features and iconography. AI-based translations for map labels and the development of 3D geospatial map data for the metaverse are also mentioned as future directions.

1/31/2023

This post details the re-architecture of Meta's asynchronous computing platform ('Async tier') to support five-fold growth and enable integration with multiple event sources. Key changes include decoupling storage reading logic from the transport layer, introducing data-source-agnostic 'schedulers' and 'tailers' for external data sources, and transitioning from a pull-based to a push-based model. This addresses challenges with system complexity and external data source integration, improving efficiency by eliminating data duplication in FOQS and enabling first-class support for various data sources.

11/21/2022

This post details Meta's implementation of Precision Time Protocol (PTP) to achieve nanosecond-level time synchronization across its systems, a significant upgrade from the millisecond-level precision of NTP. It outlines the architectural components (PTP rack, network, client), the case for PTP (improving linearizability and performance in distributed systems), and the technical innovations involved, including GNSS-over-fiber technology, a diversified Time Appliance with FPGA-based Time Cards and disciplined Network Cards, and the development of ptp4u for high-scalability PTP serving.

11/15/2022

This post introduces Sapling, a new Git-compatible source control client developed at Meta, emphasizing usability and scalability for massive repositories. It highlights Sapling's features like smartlog for repository visualization, simplified mistake recovery, and robust commit stack management. The post also announces the open-sourcing of the Sapling client and previews future plans for open-sourcing the Sapling server and virtual file system, along with the ReviewStack demonstration tool for stack-oriented code reviews.

9/19/2022

This post details the development of a new data ingestion infrastructure and last-mile transformation pipelines to support the growing demands of AI model training at Meta. Key technical contributions include: 1. Designing a disaggregated Data PreProcessing tier (DPP) for scalable data ingestion and feature transformation. 2. Implementing feature flattening as a new storage format to improve selective feature reading efficiency by 2-2.3x. 3. Introducing DPP client-side rebatching to support larger batch sizes, leading to 20-40% improvements in reader node throughput. 4. Discussing optimizations for memory bandwidth utilization in data reading workloads, including the FlatMaps project.

9/7/2022

This post introduces TAOBench, an open-source benchmark designed to accurately simulate the end-to-end social graph workload at Meta. It addresses the lack of realistic benchmarks for large social network applications by capturing transactional requirements, data colocation preferences, request distributions, and multitenant behavior, all derived from Meta's TAO graph data store. The post details the benchmark's methodology, use cases at Meta for testing new features and optimizations, and reports results on five popular distributed database systems, making Meta's social graph workload accessible to the wider database community.

9/7/2022

This post introduces 'Network Entitlement,' a contract-based network sharing solution designed to address challenges in managing Meta's growing network usage and traffic volume. It details the framework's five key properties: isolation for reliability, guarantee and accountability, abstraction of network complexity, observability, and work-conserving behavior. The post also outlines the four components of the solution: contract abstraction, dynamic SLO-based guarantee service, runtime enforcement system, and policy verification using production drills. This represents a significant step in proactively managing and scaling network infrastructure to meet increasing demands and ensure service-level objectives.

9/6/2022

Introduced Global Reservations Service and Regional Fluidity systems to manage capacity across a growing global data center footprint. Global Reservations Service simplifies capacity management for latency-tolerant services by providing declarative, intent-based global reservations that automatically manage disaster-readiness buffers and optimize placement using a mixed-integer linear programming solver. Regional Fluidity addresses latency-sensitive services by modeling demand sources and enabling safe movement of services towards a globally optimized state.

7/25/2022

This post highlights Meta's active role in advocating for the discontinuation of leap seconds, a practice that has historically caused significant disruptions and outages in digital systems. It details the technical challenges associated with leap seconds, including the complexities of 'smearing' them to mitigate negative impacts, and emphasizes Meta's support for an industry-wide effort to move beyond this outdated timekeeping mechanism to ensure greater system stability and reliability.

7/14/2022

Introduced Owl, a new system for high-fanout distribution of large data objects (executables, code artifacts, AI models, search indexes) within Meta's private cloud. Owl addresses the challenges of fanout, size, and hotness of content distribution by employing a decentralized data plane and a centralized control plane. This split design allows for efficient streaming of data from sources to clients via ephemeral, per-data-chunk distribution trees. Owl has scaled to distribute over 700 petabytes of data per day to over 10 million clients, improving download speeds and cache hit rates by 2-3x compared to previous systems like BitTorrent. The system emphasizes customizability through modular tracker interfaces for defining caching and fetching policies, and uses trace-driven emulation for policy optimization.

6/20/2022

This post introduces Transparent Memory Offloading (TMO), Meta's data center solution for managing memory costs and power consumption. TMO offloads colder memory pages to cheaper storage technologies like compressed memory and NVMe SSDs. It utilizes a new Linux kernel mechanism that measures lost work due to resource shortages (CPU, memory, I/O) in real-time via Pressure Stall Information (PSI). A userspace agent, Senpai, uses PSI feedback to dynamically adjust the amount of memory offloaded, without application modification. TMO has been in production since 2021, saving 20-32% of memory per server across millions of machines and has had its OS components upstreamed into the Linux kernel.

6/8/2022

This post introduces Zelos, a new ZooKeeper API built on the Delos distributed system platform. It addresses the scaling limitations of Apache ZooKeeper by providing a more modular, extensible, and performant solution for durable storage, distributed leases, and coordination primitives. The post details the challenges of mapping ZooKeeper's session management and consistency models to Delos and outlines Zelos's solutions for session ordering, session-based leases, and transparent migration.

3/7/2022

This post details improvements to LogDevice's read availability by modifying its Flexible Paxos consensus algorithm. The core issue addressed is the trade-off between read quorum and leader election quorum in Flexible Paxos, which previously led to 'stuck recovery' scenarios and read availability loss during leader failover. The solution involves decoupling read quorum from leader election quorum, allowing reads to proceed even when a full read quorum is not met, by leveraging LogDevice's recovery process and external consensus mechanisms like ZooKeeper to manage acceptor set reconfigurations and ensure data consistency.

11/9/2021

This post announces two new milestones for Meta's data centers: the sharing of its next-generation network hardware portfolio (Wedge 400/400C TORs and Minipack2/Arista 7388X5 fabric switches) and the migration of its data center network hardware to the Open Compute Project (OCP) Switch Abstraction Interface (SAI). It highlights the continued commitment to openness and disaggregation, building on a decade of data center design and founding OCP. The post emphasizes how these advancements in open networking hardware are laying the groundwork for future technologies like the metaverse and advanced AI applications.

9/13/2021

Introduced Superpack, a novel compression technique that combines compiler analysis with data compression to reduce the size of mobile applications. Superpack achieves over 20% savings compared to default Android Zip compression by applying advanced LZ parsing and entropy coding techniques informed by compiler analysis of code structure (ASTs, opcodes). It also transforms unstructured data into programs for compression. Superpack is applied to Dex bytecode, ARM machine code, and Hermes bytecode.

8/18/2021

This post introduces RAMP-TAO, a new protocol that enhances Meta's online social graph store (TAO) by adding atomic transaction capabilities. It addresses the challenge of providing stronger transactional guarantees on an eventually consistent, massive-scale data store, improving developer experience by simplifying transaction handling and maintaining system reliability and performance. The protocol is designed to be efficient, with minimal memory overhead and a high success rate for reads, and its layered approach allows for opt-in usage, minimizing impact on existing workloads.

8/11/2021

This post details the development and open-sourcing of the Open Compute Time Appliance and the associated Time Card. The Time Card is a PCIe card integrating a GNSS receiver, a miniaturized atomic clock (MAC), and an FPGA to provide nanosecond-level time interpolation. It addresses the limitations of off-the-shelf time appliances by offering open-source software, user-serviceable hardware, and improved security. The Time Appliance, when combined with an open-sourced time server, significantly enhances timekeeping accuracy in data centers, improving infrastructure management and distributed database performance.

7/29/2021

This post introduces a linear programming approach to optimize feature selection for ML models, addressing infrastructure capacity constraints. It frames the problem of maximizing model performance while managing limited resources (memory, CPU, storage) as a system of linear equations. The post details how to model feature gains, costs, and compression trade-offs, and provides a Python code example using NumPy and CVXPY to solve these optimization problems. It also demonstrates how this framework can be used to analyze the impact of infrastructure changes on model gain and inter-service dependencies, thereby automating feature approval and infrastructure investment decisions.

7/15/2021

Introduced Fully Sharded Data Parallel (FSDP) as a new tool for large-scale AI model training. FSDP shards model parameters, gradients, and optimizer states across data parallel workers, reducing GPU memory requirements and improving computational efficiency through communication and computation overlapping. The post details the technical implementation of FSDP, explaining how it decomposes all-reduce operations into reduce-scatter and all-gather, and provides usage examples for language models, computer vision models, PyTorch Lightning, and the FairScale library directly.

2/23/2021

This post contributes to the Scalability and Large-Scale Systems Engineering thread by detailing the research and best practices for detecting and remediating silent data corruption at scale. It provides a real-world example of data corruption in a data center application, illustrating the debug methodology and the importance of hardware resiliency, production detection, and fault-tolerant software architectures for maintaining data integrity in large-scale systems.

12/17/2020

This post introduces the concept of diversified recommendations in recommendation systems, moving beyond purely predictive accuracy to consider factors like preference broadening. It uses Polya and balancing process models to illustrate self-reinforcing and self-balancing dynamics in user preferences. The post details the complexities of real-world preferences (multidimensional, soft, contextual, dynamic) and outlines practical methods for diversification, including author, media type, and semantic diversity. It also discusses exploring similar semantic nodes in content graphs, maintaining long and short-term preferences, utilizing explore-exploit tradeoff algorithms (Epsilon-greedy, Upper confidence bound), qualitative trade-off computation, and negative guardrails based on user feedback. The core technical contribution lies in proposing a framework for balancing engagement and diversity in recommendation systems by understanding and addressing the nuances of user preferences.

12/10/2020

This post details the design and implementation of Instagram's 'Suggested Posts' feature, which aims to provide personalized content recommendations that feel like an extension of the user's home feed. It introduces a two-stage system for candidate generation (using embeddings-based similarity and co-occurrence analysis) and candidate selection (ranking posts based on engagement signals and user value models). The post also addresses the cold start problem with fallback graph exploration and popular media, and outlines strategies to ensure suggested posts align with the 'Feels Like Home' principle by prioritizing home feed sources, maintaining freshness, and balancing media types.

11/18/2020

This post introduces FioSynth, a new tool for automating storage workload suites and parsing results. It synthesizes production I/O traces to simulate diverse Facebook production services, focusing on latency outliers for realistic performance prediction and tuning. FioSynth is open-sourced to help storage device manufacturers optimize drives for hyperscale applications and standardize flash preconditioning methodology.

11/11/2020

This post details the development and adoption of Twine, Meta's cluster management system, which has enabled the creation of a ubiquitous shared infrastructure. Key contributions include scaling a single control plane to manage one million machines, supporting workload-specific customization through host profiles and the TaskControl API, and optimizing hardware usage with power-efficient small machines. The post highlights the significant increase in shared infrastructure adoption, moving from 15% to 56% of the fleet within a year, and outlines the strategy to achieve full adoption for compute services.

10/22/2020

This post details the engineering efforts to scale live streaming for millions of concurrent viewers, supporting both user-generated content and broadcast-quality streams. Key contributions include: building new ingestion capabilities with redundancy and failover for broadcast-quality streams, developing novel request coalescing and cache sharding techniques to mitigate the thundering herd problem during massive delivery fanout, and increasing local interconnection capacity with ISPs to improve viewer experience. The post also discusses future explorations into LTE-Broadcast for last-mile delivery challenges and optimizing for a large number of simultaneous smaller-reach streams for events like New Year's Eve.

10/14/2020

Introduces resctl-demo, a tool for simulating system resource conflicts to test and develop strategies for resource control and optimization in server deployments. It provides an AMI and USB drive image for easy deployment and aims to improve reliability, responsiveness, and utilization by allowing developers to build intuitive working knowledge of resource control.

9/14/2020

Introduced throughput autoscaling for Facebook.com's web tier, moving from static sizing to dynamic capacity adjustments based on estimated workload and disaster scenarios. This system uses ML models to predict steady-state and disaster demand, and load testing to determine throughput supply, leading to improved resource utilization and cost savings. The approach models demand in terms of throughput metrics (e.g., RPS) rather than utilization metrics, allowing for more accurate capacity planning even for unobserved workloads.

9/8/2020

This post details Meta's strategy for optimizing workload placement to improve fault tolerance in data centers. It introduces the concept of fault domains and the need for buffer capacity to mitigate sub-data center failures. The post outlines a multi-level approach to achieve optimal placement: ensuring hardware is spread evenly across fault domains using integer linear programming, spreading services and workloads across this hardware, and ensuring data shards for stateful services are also distributed. It discusses challenges like changing demand and evolving data center designs, and the use of physical rack moves to rebalance older capacity. The core technical contribution is the systematic approach to achieving optimal placement at hardware, service, and data levels to enhance resilience.

8/31/2020

This post details the migration of all ZooKeeper clusters from bare-metal to Meta's internal container orchestrator, Twine. It describes the challenges of running a stateful service like ZooKeeper on a stateless platform (Twine's Twine Shared Pool - TSP), the development of a task controller to enforce ZooKeeper's durability and quorum requirements, and the engineering of escape hatches to prevent circular dependencies with other Twine-powered infrastructure components.

8/17/2020

This post details the evolution and re-architecture of Meta's Async system to handle billions of asynchronous requests at scale. Key technical contributions include the introduction of 'delay tolerance' for granular prioritization of requests, capacity optimization strategies such as multi-queueing (replacing a single MySQL table), time shifting (predictive and deferred compute), and batching to reduce load on core components. The system moved from a simple first-come, first-served prioritization to a more sophisticated approach managing different traffic types (daily, major events, incident response) and ensuring fair resource allocation through policies.

8/5/2020

Introduced pcicrawler, a Python-based command-line interface tool for debugging PCI/PCIe issues at scale. It provides detailed hardware information, visualizes PCI topology, and offers machine-parsable JSON output for automation, improving upon legacy tools like lspci. It is designed to be lightweight, efficient, reliable, and safe for production environments.

2/6/2020

This post details advancements in ZooKeeper at Facebook, including a 10x improvement in write throughput through snapshot scheduler and batch commit optimizations, a new lock API for semaphore optimization reducing server round trips, demand control for multitenancy to mitigate noisy neighbor issues, zero-downtime ensemble migration for scalability, and the successful migration of the entire ZooKeeper fleet to an in-house containerization system, simplifying fleet management and deployments.

12/20/2019

This post from the @Scale Tel Aviv 2019 conference showcases Meta's continued focus on large-scale systems engineering. It details advancements in scaling data center infrastructure, optimizing testing with mutation testing, managing data prefetching for performance, adapting system designs to hardware changes, developing a new Go ORM, performing memory analysis at scale for Facebook Lite, aligning data at scale, managing monorepos, operating low-latency fraud prevention systems, and enhancing detection and alerting frameworks for metric movements. It reinforces the ongoing commitment to tackling complex engineering challenges at massive scale.

10/24/2019

This post details the award-winning work on the "Runtime Support for Multicore Haskell" paper, which significantly improved Haskell's performance on parallel hardware by addressing bottlenecks in the GHC runtime system. This resulted in up to 8x speedups for benchmarked programs, making Haskell suitable for large-scale production systems like Facebook's abuse protection system, Sigma. The work has also influenced systems in pharmaceuticals, finance, and biotech, and contributes to ongoing efforts in source code analysis.

10/17/2019

This post summarizes technical discussions from Video @Scale 2019, highlighting advancements in video streaming, encoding (including AV1, VP9, AVC trade-offs), HDR video pipelines, live streaming latency, load testing for resilience, and ML-driven video content integrity. It also touches on ML for contextual video ad safety and video analysis platforms for large corpora.

8/9/2019

This post details advancements in performance engineering for large-scale systems, including the Facebook app's journey with performance metrics, AI workload performance analysis on accelerated platforms, scaling ML models on TPUs and GPUs, and optimizing perceived performance through behavioral analytics. It also highlights open-source browser contributions for web app performance, the FlameScope tool for profiler analysis, real user monitoring for native apps, and scalable approaches to generating iOS binary order files for startup performance optimization.

6/6/2019

This post details the evolution and architecture of Twine, Meta's cluster management system. Key contributions include the development of a TaskControl interface for seamless stateful service management, enabling stateful applications to influence container lifecycle operations for data availability. It also describes the evolution towards a single control plane across data centers, facilitated by Resource Broker, to manage servers dynamically and simplify cluster decommissioning and maintenance. The post highlights the shift to a large global shared fleet of servers and the challenges of scalability and competing requirements, leading to transparent sharding of the control plane and an elastic compute approach for real-time capacity shifting.

5/28/2019

This post details the evolution of Meta's DHCP infrastructure, moving from ISC-DHCP to ISC Kea, and finally to an extended DHCPLB that functions as a server. The primary technical contribution is the replacement of the single-threaded Kea DHCP server with a multithreaded DHCPLB server, built upon a new Go DHCP library. This change addresses performance bottlenecks caused by Kea's single-threaded nature and the blocking Recv-Q buffer, leading to significantly improved throughput, reduced server count (10x fewer servers for the same traffic volume), and faster iteration cycles. The post also discusses the challenges of previous architectures, the benefits of the new Go DHCP library, and the use of DHCPLB for A/B testing and phased rollouts.

11/6/2018

This post introduces Getafix, a novel tool that automatically finds and suggests fixes for software bugs by learning from past engineer code changes. It enhances code quality and system stability by integrating with existing testing tools like Sapienz and Infer, and represents a significant advancement in automated bug remediation at scale.

10/30/2018

This post details the open-sourcing of several Linux kernel components and tools developed by Facebook to address production issues in large-scale cloud computing. Key contributions include BPF for flexible kernel behavior modification (used in the Katran load balancer), Btrfs as a next-generation filesystem for fault tolerance and administration, Netconsd for lightweight log transport, Cgroup2 for workload grouping and resource control (driving multi-tenancy and efficiency), PSI for quantifying resource shortages, and Oomd as a userspace OOM killer for proactive memory management. These tools are in production on millions of servers and aim to improve resource utilization, reliability, and scalability.

10/19/2018

This post introduces MobileLab, a new system designed for highly accurate mobile performance testing. It addresses the challenge of detecting very small performance regressions (as low as 1 percent) in highly optimized applications at scale. MobileLab achieves this by establishing a deterministic testing environment, eliminating variance through techniques like fixing CPU/GPU frequencies, using an HTTP proxy for consistent network requests, managing application state across trials, and optimizing disk I/O and device clock synchronization. This significantly improves confidence intervals and reduces false positives, preventing performance regressions from shipping.

10/8/2018

This post introduces Akkio, a novel data placement service (DPS) designed to manage data store locality at scale for geo-distributed systems. Akkio optimizes data retrieval speed by intelligently determining where to store data replicas, reducing duplication, WAN traffic, and latency. It achieves this by splitting data into microshards, analyzing access patterns, and considering regional capacity to dynamically migrate data. This represents a significant advancement in managing data at Facebook's scale, complementing existing distributed storage and data management strategies.

10/3/2018

This post details the AI-powered systems behind Facebook Marketplace, focusing on product indexing, retrieval, and seller assistance. Key technical contributions include: a multimodal ranking system combining Lumos (image understanding) and DeepText (text understanding) for product indexing; the use of FAISS for efficient similarity search to reduce computing load and speed up retrieval; distributed training of neural network models across multiple machines to reduce training time from over a week to 1-2 days; optimizations to increase QPS and CPU utilization by tuning classification threads and improving memory purging configurations (MALLOC_CONF='opt.decay_time:-1'); AI-driven autosuggestions for seller categories and pricing; and the use of M (an AI assistant) in Messenger for automated suggestions, including replies, product recommendations, and message translations via neural machine translation. The post also mentions ongoing testing of image-based search for visually similar products.

7/19/2018

This post introduces oomd, a new userspace out-of-memory (OOM) killer developed by Facebook. It addresses limitations of the traditional Linux OOM killer by providing faster, more reliable, and customizable OOM handling. Key contributions include the use of Pressure Stall Information (PSI) and cgroup2 for proactive monitoring and a plugin system for custom OOM response strategies, significantly reducing livelocks in large-scale infrastructure.

7/19/2018

This post introduces Location-Aware Distribution (LAD), a new peer-to-peer system designed to address the scalability challenges of distributing configuration changes to millions of servers. LAD improves upon the previous ZooKeeper-based system by supporting much larger configuration files (100 MB vs 5 MB), increasing the number of subscribers per distributor (40,000 vs 2,500), and separating data storage from the distribution framework. It utilizes a tree-based peer-to-peer network with sharding to manage updates efficiently and reduce latency, while also highlighting lessons learned in tooling, monitoring, and failure testing for large-scale distributed systems.

7/13/2018

Introduced XARs (Executable Archives) as a system for distributing self-contained executables that encapsulate data and code dependencies. XARs leverage a FUSE-based filesystem and squashfs for on-demand decompression and efficient caching, aiming to improve deployment speed, robustness, and efficiency compared to previous methods like PARs and PEX files. Benchmarks show improvements in file size and start times, particularly for short-running executables. XARs are open-sourced via GitHub and PyPI.

6/28/2018

Introduced Spiral, a system for self-tuning high-performance infrastructure services at Facebook scale using real-time machine learning. Spiral replaces hand-tuned heuristics with machine learning classifiers, enabling services to optimize themselves in minutes by learning from real-time feedback. This system integrates as a small, embedded C++ library with prediction and feedback call sites. It can operate in an embedded mode or send data to a backend service for heavy lifting in training and model selection. A case study demonstrates Spiral's application in automating reactive caching heuristics, where it learns to classify database updates as relevant or irrelevant to query results, significantly reducing manual engineering effort and improving adaptability.

6/19/2018

Introduced BOLT, a binary optimization and layout tool that addresses instruction starvation by optimizing instruction placement in memory. BOLT reconstructs control flow graphs, uses LLVM's MCInst format, and supports Linux binaries (x86-64, AArch64). It leverages sample-based profiling from Linux `perf` tool. BOLT achieved 3-8% performance improvement on HHVM and 2-15% for other services, by isolating frequently executed code and optimizing for hardware features like macro-fusion. BOLT also performs advanced jump table placement, identical code folding, PLT optimization, and constant load elimination.

6/12/2018

This post details the integration of data science into infrastructure engineering at Meta, focusing on three key areas: optimizing CDN cache efficiency with ML models to predict content re-request likelihood; developing tools to detect performance regressions by isolating effects from underlying trends in device usage or demographics and using custom algorithms for server-side regression detection that account for trend, seasonality, and noise; and improving data center strategy through custom forecasting models (Bayesian time series, deep learning) that handle complex seasonal fluctuations and usage trends, and predicting the impact of infrequent events on infrastructure.

3/20/2018

This post introduces the Fabric Aggregator, a novel distributed network system designed to scale data center network capacity. It details a disaggregated approach using simple, open building blocks like Wedge 100 switches and FBOSS, enabling flexible adaptation to varied traffic patterns and future growth. The implementation of a two-layer cross-connect architecture (downstream for east/west traffic and upstream for north/south traffic) allows for independent scaling of capacities. The post also discusses operational aspects, redundancy, and different deployment types (single-rack and multi-rack) with various cabling configurations, emphasizing efficiency, power savings, and increased flexibility.

3/19/2018

This post recaps the second annual Performance @Scale conference, highlighting discussions on managing performance challenges for large-scale services. Key contributions include insights into anomaly detection for real user monitoring, the open-source release of Profilo for mobile performance tracing, advancements in distributed trace processing for performance regression detection (Execution Graphs), iOS VM and loader considerations for app startup time, mobile performance testing in real user conditions, global network emulation, scaling real-time infrastructure for major shopping events, and efficient static resource delivery mechanisms for the web platform. It underscores the ongoing focus on performance engineering as a critical aspect of scalability.

2/12/2018

This post details how Meta's Production Engineering team supports global events on Facebook, specifically focusing on the planning and infrastructure required for Facebook Live during New Year's Eve. It outlines the different types of load variance (routine, spontaneous, planned), the architecture of Facebook Live, key load metrics (total broadcasts, peak concurrent broadcasts, load on dependent systems), and the process of estimating and scaling for peak traffic. It also describes load testing methodologies (artificially increasing load, synthetic load tests, shadow traffic) to ensure system reliability during these high-traffic periods.

1/23/2018

Introduced a scalable solution for merging Android native libraries to overcome the pre-4.3 Android native library limit. This involved integrating the feature into the Buck build system, developing a strategy to handle the duplicate JNI_OnLoad symbol using custom ELF sections and SoLoader, and providing a demonstration repository for the techniques.

10/18/2017

This post summarizes the Mobile@Scale 2017 conference, showcasing Meta's and other industry leaders' advancements in scaling mobile products and services. Key contributions include discussions on multi-modal interfaces, WhatsApp's infrastructure evolution within Facebook's data centers, multi-threaded rendering on Android using Litho & Infer, modern server-side rendering with React, real-time analytics for billions (Fabric), scaling mobile apps like Runkeeper and Instagram, Android development processes at Facebook, zero power voice activation, scaling Recommendations, high-performance app development and bottleneck analysis, battery instrumentation for Facebook apps, and travel tech at scale with Hopper. It reinforces the ongoing commitment to mobile scalability within the broader @Scale initiative.

8/31/2017

This post introduces LogDevice, a new distributed data store specifically designed for logs. It addresses the challenges of reliably storing ordered immutable records at Facebook's scale, focusing on high availability, durability, and repeatable total order. LogDevice's design decouples record sequencing from storage, uses non-deterministic record placement for improved availability and load balancing, and employs epochs for sequence numbers to handle sequencer failures. This contributes to the broader theme of scalable and robust data storage solutions within Meta's infrastructure.

8/28/2017

Introduced FBNet Command Runner (FCR), an open-source command runner service with a Thrift interface for running commands on network devices. FCR abstracts vendor-specific details, handles multiple device vendors and addresses, and scales to tens of thousands of devices, simplifying network monitoring, provisioning, and deployment. It integrates with the Robotron system for active monitoring.

6/15/2017

This post recaps the 3rd annual Data @Scale conference, highlighting discussions on building large-scale storage systems and analytics, the impact of Big Data on machine learning, and the evolution of infrastructure and data center design. It features presentations on accelerating machine learning for computer vision, next-generation globally-distributed databases, real-time analytics DBMS, machine learning serving platforms, micro-service architectures, data warehousing and experimentation platforms, globally distributed data management systems, cloud specialization architectures, and bulk data movement for global data storage. The post also mentions a Women in Engineering Breakfast & Panel, emphasizing community and collaboration in addressing these challenges.

6/5/2017

This post, a recap of the Dev Tools @Scale 2017 conference, highlights Meta's contributions and engagement in developing and utilizing advanced developer tools to enable large-scale development. It specifically showcases internal projects like Project OneWorld for resource management and testing, Infer for static code analysis, and Jupiter for job scheduling, alongside industry-wide discussions on code search, source control scaling, and build systems.

5/24/2017

Introduced One World, a unified resource management system for hosting mobile devices, web browsers, and emulators for large-scale testing. The system architecture includes runtime worker services, a One World daemon, the Jupiter scheduler, and Satellite mode for local resource integration. Key technical details include state machine-based resource lifecycle management, health checks, state restoration for mobile devices, and remote access mechanisms for Android (adb tunnels) and iOS (FBSimulatorControl, WebDriverAgent). The system manages tens of thousands of resources and executes over 1 million jobs daily.

5/24/2017

Introduced Jupiter, a high-performance job-matching service for distributed systems. Jupiter matches jobs to workers based on multidimensional capabilities (e.g., hardware resources, pre-loaded data, kernel version) and guarantees atomic job acquisition. It is horizontally scalable, written in C++ with Thrift, and can handle hundreds of thousands of requests per second per shard with tens of microseconds latency for work acquisition. Jupiter decouples job storage from matching, acting as an arbiter rather than a persistence layer.

5/5/2017

This post announces the rebuilding of the OnlineSchemaChange tool in Python (OSC.py), enhancing its ease of use with a standalone CLI, improved testability through a robust testing framework, and increased reliability with data consistency checks. It also outlines future plans for triggerless schema changes and utilization of native online DDL.

4/18/2017

Introduced Relay Modern, a new version of the Relay framework. Key improvements include: simplified API, enhanced performance through static queries and ahead-of-time optimization, simpler mutations with explicit field fetching and cache update specifications, and built-in garbage collection for cache eviction. The framework is now modularized into a compiler, runtime, and React integration layer, increasing extensibility. An iterative adoption path with a compatibility API is provided for existing Relay applications.

3/27/2017

This post introduces optimizations for gradient-boosted decision trees (GBDTs) in C++ to improve evaluation efficiency for large-scale ranking systems. It compares naive flat tree implementations with compiled trees, leveraging LIKELY/UNLIKELY annotations for branch prediction. Further improvements include range-based evaluation to optimize CPU cache usage by evaluating samples across tree ranges, and techniques for handling common and categorical features to reduce model size and evaluation time. Computational results show up to a 5x performance improvement over the flat tree implementation.

1/24/2017

This post details Meta's contributions to and the growing ecosystem around open networking hardware, including the open-sourcing of Wedge 100, Backpack, and Voyager. It highlights collaborations with companies like Cumulus Networks, Big Switch Networks, Barefoot Networks, SnapRoute, Canonical, Cavium, and Apstra to integrate various network operating systems (Cumulus Linux, ONL, OpenNSL, FBOSS, SONiC, Ubuntu Core, FlexSwitch) and software components with this open hardware. It also announces the commercial availability of Wedge 100 and Backpack, and the development of new hardware variations like Wedge 100B and Wedge 100C.

12/8/2016

This post details the engineering behind Facebook's 'Recommendations' feature, which extracts place suggestions from free-text comments. It describes the use of conversational understanding (CU) and local search technologies to identify and resolve place entities from unstructured text. The post highlights the technical challenges of dynamically updating feed stories across platforms (Android and iOS) with place cards and map pins, including manual client-side caching strategies and the use of ComponentKit for dynamic UI rendering on iOS. It also outlines the tech stack for entity retrieval and scoring, involving Gaussian mixture models and LambdaMART algorithms.

10/31/2016

This post documents the Networking @Scale Boston conference, highlighting discussions on cloud networking, traffic configurations, content delivery network infrastructure, and the scaling of applications and services. Key contributions include insights into Facebook's Open/R routing platform, scaling Facebook Live, traffic configuration evolution, and securing network pipes. It also features discussions from other industry leaders on Google's networking efforts, Akamai's cloud networking, browser networking performance, and QUIC protocol adoption.

10/19/2016

This post details a comparative study of Apache Giraph and Spark's GraphX for large-scale graph processing at Facebook. It quantifies performance and scalability differences, finding Giraph to be significantly more efficient for production-scale workloads (processing 50x larger graphs and requiring less memory), while noting GraphX's advantages in usability and easier development for certain tasks. The post also outlines the methodology, experiment setup, and key findings regarding performance, scalability, and usability.

9/13/2016

This post details the development and productionization of DHCPLB, an open-source DHCP load balancer designed to address traffic imbalance issues in Facebook's production DHCP infrastructure. It covers the identification of the problem (uneven traffic distribution due to anycast and network topology), the hackathon proof-of-concept, and the intern-led project that added DHCPv6 support, A/B testing capabilities for KEA servers, MAC address-based overrides, integration with monitoring infrastructure, hot-reloading configuration, BGP advertisement via ExaBGP, and two balancing algorithms (RoundRobin and ModuloN). The post also outlines the architecture and deployment of DHCPLB within Facebook's data centers.

9/1/2016

This post announces new technologies presented at the @Scale 2016 conference, highlighting advancements in data compression (Zstandard), database storage engines (MyRocks), 360 video stabilization, and developer tools for automated regression detection (BrowserLab) and static analysis (Infer). These contributions showcase Meta's ongoing efforts to address large-scale engineering challenges and improve efficiency across various domains.

8/31/2016

This post introduces BrowserLab, a system designed to automatically detect and prevent client-side performance regressions in web applications. It details the challenges of measuring client performance in a highly interactive web environment, the sources of noise in performance testing (environment, data, code non-determinism), and the system's design to mitigate these issues through HTTP proxies, backend proxies, and deterministic code execution. BrowserLab's ability to detect regressions as small as 20ms and its integration into the continuous integration pipeline for automated bisecting and alerting are highlighted as key contributions to maintaining site performance at scale.

8/31/2016

This post details Meta's experience migrating a 60 TB+ batch analytics pipeline for entity ranking from Hive to Apache Spark. It highlights the challenges of the previous sharded Hive implementation and the benefits of a unified Spark job. The post extensively covers the reliability fixes (handling node reboots, unresponsive drivers, OOM errors) and performance improvements (memory leak fixes, Snappy optimizations) required to scale Spark to this unprecedented level, including contributions back to the open-source Spark project.

8/31/2016

Introduced Zstandard (Zstd) 1.0, a new general-purpose lossless compression algorithm and implementation. Zstd offers significant improvements in compression and decompression speeds compared to Deflate (used in Zip, gzip, zlib) while achieving similar or better compression ratios. Key technical innovations include the use of Finite State Entropy, a performance-first design optimized for modern CPUs, a branchless design to avoid pipeline flushes, and a format designed for parallel execution by separating data into multiple parallel streams. Zstd offers 22 compression levels for granular trade-offs between speed and size, and its memory usage is more flexible than zlib's 32 KB window.

8/24/2016

This post introduces Robotron, a top-down network management system designed to manage Facebook's massive production network of tens of thousands of devices and hundreds of thousands of servers. It details the system's evolution from initial monitoring efforts (FBNet) to a comprehensive solution covering network design, configuration generation, deployment, and monitoring. Key challenges addressed include distributed configurations, handling multiple domains, versioning, dependencies, and vendor differences. The system translates high-level operator intent into vendor-specific configurations and deploys/monitors them to ensure network state consistency. The post also highlights the open-sourcing of parts of Robotron, such as fbpush.

8/3/2016

This post details the evolution of Meta's hardware engineering practices, moving from isolated labs to a collaborative environment with the creation of 'Area 404'. It highlights the investment in state-of-the-art machinery for in-house modeling, prototyping, and failure analysis, significantly reducing development cycles. The post showcases how this collaborative approach accelerates innovation across infrastructure (e.g., Wedge, 6-pack, Open Vault), Connectivity Lab (e.g., Aquila, Terragraph), and VR (e.g., Surround 360, Oculus) projects.

6/17/2016

This post details the growth of the Open Compute Project (OCP) networking community, highlighting the development and adoption of open-source network switches like Wedge and Wedge 100. It showcases how disaggregating hardware and software in networking devices has enabled a flexible, scalable, and efficient ecosystem, with contributions from various companies and startups experimenting with and building upon these open platforms.

6/16/2016

This post recaps the Data @Scale conference in June 2016, showcasing discussions and presentations on exabyte-scale storage systems (Dropbox), integrated analytics for storage (Qumulo), MPP shared-nothing databases on flash (Presto Raptor by Facebook), lessons from studying modern databases and key-value stores (University of Wisconsin-Madison), flexible consistency levels in distributed key-value stores (Manhattan by Twitter), scale-out resource management (YARN++ by Microsoft), a unified model for batch and streaming data processing (Apache Beam by Google), data integration for visual analysis (Tableau), and disaggregated storage and compute for Map/Reduce (Facebook). It emphasizes the collaborative nature of the event and the sharing of practical insights for building and operating high-scale systems.

5/27/2016

This post details the extensive and foundational use of Python in Facebook's production engineering, highlighting its role in infrastructure management (hardware lifecycle, network setup, fault remediation, maintenance scheduling), platform services (job scheduling, binary distribution, TFTP, workflow execution), service configuration management (dynamic configuration generation), and operational efficiency (database health monitoring, phased rollouts). It quantifies Python's impact with statistics on codebase percentage, commits, and committers, and discusses the ongoing transition to Python 3 for performance gains.

4/20/2016

This post introduces Wangle, a C++ library for asynchronous networking and RPC, heavily influenced by JVM libraries Netty and Finagle. It details Wangle's core components, including asynchronous building blocks from Folly (IO wrappers and Futures), concurrency primitives like CPU and IO thread pools with optimizations for different workloads, and a pipeline architecture for processing network data through a series of handlers and codecs. The post also covers bootstrapping servers and clients with ServerBootstrap and ClientBootstrap, and provides an example of building a file streaming server that leverages Wangle's capabilities to offload blocking disk reads to a CPU thread pool.

4/16/2016

This post announces the open-sourcing of 14 core Chef cookbooks developed and used internally at Meta. These cookbooks have been refactored to be generic and allow external users to adopt Meta's configuration management model. The post details the history of Meta's involvement with Chef and the motivation for releasing these cookbooks. It provides a quick start guide and demonstrates how to use the cookbooks with examples of setting sysctls, limits, logrotate, swap, modprobe, and fstab. It also explains the hierarchical configuration model enabled by these cookbooks and delves into the specifics of the `fb_fstab` cookbook, highlighting its advanced features for managing `/etc/fstab` and mounted filesystems.

3/30/2016

This post details the engineering efforts behind the "On This Day" feature's evolution, focusing on user experience research, filtering mechanisms, and a machine-learning-based ranking system. It highlights the use of automatic filters to exclude unwanted memories (e.g., with exes, blocked individuals) and the implementation of user preferences for blocklisting specific people or dates. The ranking model is described as being trained in real-time using personalization signals (user interactions, demographics, memory attributes) and content understanding from a computer vision platform that leverages deep convolutional neural networks to recognize visual concepts within photos and videos. This enables a more personalized and relevant memory surfacing experience.

3/18/2016

This post introduces Dragon, a distributed graph query engine designed to handle complex, multi-hop queries on Facebook's social graph. It builds upon the existing TAO system by introducing advanced indexing techniques, including socially aware inverted indices and optimized data layouts, to significantly improve the efficiency of fetching, filtering, and reordering graph data. Dragon aims to reduce network transfer, free up CPU, and serve a larger dataset from cache, demonstrating a key step in scaling graph query capabilities.

2/29/2016

This post recaps the Performance @Scale 2016 event, showcasing industry-wide discussions and advancements in making technology fast and efficient. It highlights specific techniques and tools for performance analysis and optimization, including BPF for Linux profiling, Facebook's web speed solutions (Time-to-Interact, Display Done, Early Flush, Big Pipe, Bootloader), automatic regression triaging (AutoTriage, Stack Trace Finder, Pushed Commit Search, Diff Ranker), Google's approach to ad revenue improvement through performance analysis, Microsoft's client-side performance investigation with PerfView, Google's high-performance networking in Chromium (histograms, QUIC protocol), and Facebook's mobile performance engineering with the Loom telemetry framework, as well as LinkedIn's Real User Monitoring (RUMv2) and visualization techniques.

12/28/2015

This post details an improvement to software RAID 5/6 by introducing a write-ahead log. This addresses critical issues like the 'write hole' during power failures, significantly speeds up rebuild times after disk failures, and optimizes write performance by allowing aggregation of writes. The implementation uses an SSD as the log disk and has been merged into the Linux 4.4 kernel, enhancing the reliability of large-scale storage systems.

11/20/2015

This post details the extensive efforts undertaken to optimize the Facebook app's startup time on iOS. It outlines the process of defining stable performance metrics, distinguishing between cold and warm starts, and breaking down the cold start problem into request time, network time, and response processing time. Key innovations include UDP priming to pre-fetch data and a 1+1+X story fetching strategy to improve server-side processing and download times. The post also highlights how non-critical services were enqueued to run after initial story display, further reducing perceived latency.

11/19/2015

This post details Meta's significant advancements in open networking with the introduction and deployment of Wedge, an open-source top-of-rack network switch, and FBOSS, its Linux-based control software. It highlights the disaggregation of networking hardware and software, enabling faster innovation and customization. The post elaborates on operationalizing Wedge and FBOSS for large-scale deployments, including custom tools for seamless software updates, ensuring nonstop forwarding with Warm Boot, and lessons learned from treating switches like servers. It also discusses crucial CPU protection mechanisms and future plans for scaling this open networking approach to higher speeds and larger aggregation switches.

10/26/2015

This post introduces Meta's approach to performance instrumentation for Android apps, addressing the limitations of manual instrumentation and built-in profiling tools. It details the development of a bytecode instrumentation system that automatically injects code to collect detailed performance telemetry from real-world usage. Key contributions include the ability to trace asynchronous operations across threads and the development of a lock-free ring buffer for efficient event collection, enhancing the understanding of performance bottlenecks in a rapidly evolving mobile environment.

10/5/2015

This post details Meta's "upstream-first" philosophy for Linux kernel development, emphasizing active participation in the upstream community to integrate their specialized kernel modifications for large-scale deployments. It highlights contributions in areas like blk-mq for storage performance, Btrfs filesystem stabilization, NV congestion control, cgroup writeback support, network virtualization (GUE, ILA), IPv6 performance improvements, and RAID 5/6 caching. The post showcases the team's commitment to improving the Linux kernel for both internal use and the broader community, enabling faster adoption of new technologies and reducing internal churn.

8/14/2015

This post details the porting and feature additions to OpenBMC for the 'Yosemite' multi-node server platform. Key technical contributions include: removing the RMCP+ protocol for security, adding SSH and REST API support for management and debugging, implementing IPMI FRUID parsing for various Field Replaceable Units, developing a sensor monitoring daemon for analog sensors, updating software modules for multi-node support (including payload number in IPMI requests and IPMB framework), adding NCSI support to the Ethernet driver, enhancing the I2C driver for secondary mode and IPMB communication, adding IPMI SDR support with a caching daemon, implementing discrete signal monitoring for error conditions, and introducing new utilities (fruid-util, sol.sh, power-util, sensor-util) accessible via SSH.

7/21/2015

This post details the migration of Facebook's data center DHCP infrastructure from ISC dhcpd to ISC Kea. It highlights the challenges with the previous system, including slow configuration propagation and frequent server restarts due to static configuration files. The adoption of Kea is driven by its modern design, extensibility via hook points, and suitability for a stateless deployment within containers. The new system fetches configuration dynamically from the inventory system, simplifying maintenance and accelerating hardware provisioning.

3/10/2015

Introduced "Yosemite", an open-source modular chassis design for high-powered microservers, as a contribution to the Open Compute Project. Yosemite utilizes system-on-a-chip (SoC) processor cards targeting up to 65W TDP, with multiple memory channels, local SSD interface, and a management controller. It extends the OCP microserver interface with an additional PCI-E x16 connector and features a shared network connection for data and management traffic. The design supports up to 4 SoC cards per chassis, compatible with Open Rack, and aims to provide a cost-effective, flexible, and easy-to-service system for scale-out infrastructure.

3/10/2015

This post announces the open-sourcing of FBOSS (Facebook Open Switching System) and the contribution of the Wedge switch specification to the Open Compute Project. FBOSS is a set of applications that run on a standard Linux OS to program and control network ASICs, enabling automation, configuration, monitoring, and troubleshooting. The Wedge switch is a disaggregated top-of-rack switch designed to be managed like a server. The post details the anatomy of a network switch, explaining the role of ASICs and the FBOSS agent in programming them. It also discusses the disaggregated model of network hardware and software, contrasting it with traditional monolithic network switches. The FBOSS agent's lean feature set, focused on L3 unicast routing and ECMP, is highlighted, along with its reliance on Broadcom's OpenNSL APIs. The post outlines future plans for FBOSS development, including adding control/routing protocol code and supporting other environments.

3/10/2015

This post introduces OpenBMC, an open software framework for next-generation system management, developed in-house due to the limitations of closed BMC software stacks. It details the motivation for building OpenBMC, its "under the hood" architecture as a customized Linux distribution built with Yocto Project, and its initial deployment on Facebook's Wedge switch. The post outlines the layered structure of OpenBMC (common, SoC, and board layers) and calls for community contributions to foster an ecosystem for BMC innovation.

2/11/2015

Introduced '6-pack', the first open hardware modular switch platform, designed to address the limitations of traditional networking technologies at scale. '6-pack' is a full mesh non-blocking two-stage switch composed of 12 independent switching elements, each capable of 1.28Tbps. It leverages 'Wedge' as a building block and features a split control configuration with hybrid SDN for management. The platform includes modular line cards and fabric cards, enabling the creation of switches of any radix using common building blocks. The design is intended to be contributed to the Open Compute Project.

11/14/2014

Introduced the 'data center fabric', a next-generation network architecture for Meta's data centers. This fabric replaces the previous cluster-based design with a disaggregated approach using server pods and uniform high-performance connectivity via fabric switches and spine switches across four planes. Key technical details include the use of standard BGP4 routing with a distributed control, centralized override hybrid approach, layer3 and dual-stack (IPv4/IPv6) operation, heavy reliance on ECMP with flow-based hashing, and a multi-speed link strategy (40G inter-switch, 10G server). The design emphasizes modularity for rapid scaling of compute, network, and external connectivity, and allows for gradual scalability from an initial oversubscribed state to a non-oversubscribed network by adding spine switches. The physical infrastructure and cabling were optimized for rapid deployment.

6/18/2014

Introduced "Wedge," a disaggregated top-of-rack network switch, and "FBOSS," a Linux-based operating system for it. "Wedge" uses a modular microserver architecture, allowing for interchangeable processors (Intel, AMD, ARM) and integration into existing fleet management systems. "FBOSS" provides a Thrift-based abstraction layer for switch ASIC APIs, enabling network devices to be treated as services and facilitating hybrid distributed/centralized control for improved utilization and faster recovery. It also integrates with existing environmental monitoring tools for performance insights.

7/11/2013

Introduced Windex, an automated system for database provisioning, specifically for MySQL DB hosts. Windex handles the entire lifecycle of provisioning, from re-imaging hosts after production use or repair, to OS installation, configuration, and final checks before adding them to the spares pool. It integrates with MPS to detect hosts needing re-imaging and includes retry logic and failure pattern detection to prevent infinite loops and identify hosts requiring manual review. Robust validation checks are performed at multiple stages to prevent damage to production systems.

3/14/2013

This post details the extension of the Unicorn inverted index framework to support search ranking for Graph Search. Key contributions include: maintaining separate Unicorn verticals for different entity types, enhancing retrieval operations with weak 'and' and strong 'or' semantics, implementing query rewriting to translate user intent into structured queries, introducing scoring mechanisms (including forward index and result set scoring) to assign relevance scores to entities, developing a blending algorithm to combine results from different verticals, and extending nested query support for complex graph traversals. An A/B testing framework for ranking experiments is also described.

1/15/2013

This post details the development of Facebook's Graph Search, a new product for exploring the social graph. It covers the challenges of scaling search infrastructure for over a billion users and trillions of connections, the need to consolidate three separate search systems into a unified platform (Unicorn), and the implementation of privacy checks to ensure users only see content they have access to. The post also touches on the product's evolution from early prototypes to a beta launch, focusing on the technical aspects of building a structured search mechanism.

10/3/2012

Introduced the App Center recommendation engine, detailing its aggregator-leaf architecture for handling large-scale data and traffic. Described scaling strategies through sharding and replication. Defined a quality metric based on user ratings and daily active users. Outlined algorithmic components: candidate selection (demographics, social, past interactions), scoring (explicit, dynamic, latent features), and real-time updates for user feedback, dynamic features, and latent features derived from low-rank matrix approximation.

9/27/2011

This post introduces a glossary of software design terms including Element, Coupling, Cohesion, Succession, Socialization, Latency, Throughput, and Variance. It provides definitions and examples, particularly highlighting how coupling can lead to unexpected failures in large systems and how succession is crucial for safe data migrations. The post emphasizes the importance of managing these concepts for responsive and efficient system design.

9/8/2011

Developed an algorithm to predict business operating hours based on user check-in data. This algorithm involves making initial predictions based on an openness threshold, filling short gaps in predicted closed hours, and aggregating predictions across days. Implemented a user interface for suggesting business hours and integrated it into the Places Editor. Developed a 'check-in similarity score' to validate user-suggested hours. Added a 'Status' field to place pages to indicate current open/closed status and integrated this information into city pages.

5/16/2011

This post details the development and use of 'Project Triforce,' a simulated third data center region, to prepare Facebook's software stack for the launch of the Prineville, Oregon data center. It highlights challenges such as software complexity across hundreds of back-end services, new hardware configurations like Flashcache with MySQL, and 'unknown unknowns' in multi-region assumptions. The solution involved reconfiguring an active production cluster in Virginia to mimic a third region, using latency to stress-test the stack. The post also introduces 'Kobold,' an in-house suite for automating cluster configuration and bootstrapping, enabling rapid provisioning and deployment of tens of thousands of servers within 30 days and serving production traffic within 60 days.

12/2/2010

This post introduces the concept of a 'full-stack programmer' and emphasizes understanding how different layers of a system behave, particularly for performance and optimization. It details how data shape, working data size, request size, request rate, mutation rate, consistency, locality, computation, latency, and contention influence system design. It also breaks down the performance characteristics of different I/O types (registers, CPU cache, main memory, SSD, magnetic disk, network) and discusses the importance of minimizing I/O wait times and understanding the latency gaps between these components. The post advocates for trusting the kernel or database for data management and provides an example of estimating data transfer time based on user count, data size, and network uplink speed.

7/21/2010

This post outlines general principles for scaling systems to handle massive user growth, including horizontal scaling, moving fast, changing incrementally, measuring everything, small independent teams, and aligning control with responsibility. It highlights the technical challenges of managing 500 million active users, 100 billion hits per day, 50 billion photos, and trillions of cached objects, and discusses the importance of these principles in overcoming these challenges.

12/17/2009

Developed the 'Dyno' system for real-world web application benchmarking, measuring request duration as a function of throughput. This system allows for the comparison of hardware architectures (e.g., Harpertown vs. Nehalem) by simulating production traffic and identifying contention-induced latency. The analysis revealed that memory access contention is a primary bottleneck for web nodes, and the Nehalem/Tylersburg platform offered a 62% increase in request throughput and a 44% improvement in performance-per-watt compared to the Harpertown/San Clemente platform.

8/12/2009

Introduced a dedicated memcache tier for username availability checks, making memcache the authoritative source for this data. Implemented fault tolerance mechanisms for memcache nodes, including log replay from Scribe and a secondary memcache check for misses. Utilized optimistic concurrency control for username assignment to improve write performance. Explored and rejected Bloom Filters due to false positives and complexity. Conducted extensive load testing exceeding 10x expected launch traffic to identify and resolve infrastructure bottlenecks. Developed contingency plans ('nuclear options') for load reduction, which ultimately were not needed due to robust design.

12/12/2008

Implemented per-thread shared connection buffer pool for TCP and UDP sockets to reclaim memory. Switched to UDP for get operations and implemented application-level flow control for multi-gets. Addressed Linux UDP performance issues by using separate UDP sockets for transmitting replies per thread. Introduced opportunistic polling of network interfaces (combining interrupt-driven and polling-driven IO) to distribute network processing evenly across cores. Moved memcached stat collection to per-thread and aggregated results on-demand to eliminate global lock contention. Modified the network device transmit queue dequeue algorithm to batch dequeues, drop the queue lock, and then transmit batched packets, significantly reducing lock contention.

10/24/2008

Introduced Scribe, a unified, scalable, and reliable system for collecting billions of messages daily from servers. Key design decisions include a flexible directed graph topology, disk spooling for reliability (balancing data loss risk with performance), and a simple data model (category and message). Built using Thrift for broad language support and open-sourced to benefit the wider community.