Site Reliability and Outage Management

6/3/2026

Introduced 'Instantaneous PowerLoss Storm' testing paradigm for zero-notice power loss in data centers. Detailed strategies for building readiness into existing systems, including defense-in-depth, handling unavailability events (UEs), and addressing bootstrapping challenges like circular dependencies (with solutions like Belljar tests and a Twine recovery kit) and the 'boomerang' problem. Defined tradeoffs between reliability and velocity, identifying critical impacts to avoid (data loss, facility damage) and tolerable risks (transient errors, bounded staleness). Validated readiness through incremental testing, culminating in de-energizing large production regions.

12/19/2025

Introduced DrP, a root cause analysis (RCA) platform designed to programmatically automate incident investigation. DrP features an expressive SDK for authoring investigation playbooks (analyzers) with ML algorithms for data access and problem isolation, a scalable backend for multi-tenant and isolated execution, seamless integration with mainstream workflows like alerts and incident management tools, and a post-processing system for automated mitigation actions. It has achieved widespread adoption at Meta, reducing MTTR by 20-80% and enhancing on-call productivity.

8/6/2025

This post introduces Diff Risk Score (DRS), an AI-powered technology built on a fine-tuned Llama LLM that predicts the likelihood of code changes causing production incidents (SEVs). It highlights DRS's impact on enabling code unfreezes during sensitive periods, leading to significant productivity gains and reduced engineering time spent on incident mitigation. The post also outlines future directions for risk awareness in software development, including expanding to configuration change risk, automating risk mitigation, and improving natural language explanations for risk-aware technologies.

2/13/2025

This post highlights the significant impact and evolution of Meta's bug bounty program in 2024, awarding over $2.3 million and exceeding $20 million in total bounties since 2011. It details the program's expansion to cover emerging areas like GenAI and AR/VR hardware, the introduction of new payout guidelines for ads audience tools, and the continued efforts to build and celebrate the global security research community through events and recognition. The program's role in Meta's defense-in-depth strategy and its contribution to platform safety are emphasized.

2/5/2025

This post introduces the Automated Compliance Hardening (ACH) tool, a novel system that leverages Large Language Models (LLMs) for mutation-guided test generation. ACH automatically generates realistic faults (mutants) in source code and then creates tests specifically designed to catch these faults. This approach moves beyond traditional code coverage-focused testing by targeting specific bugs, with verifiable assurances that the generated tests will indeed catch the intended faults. ACH has been applied to critical platforms like Facebook Feed, Instagram, Messenger, and WhatsApp, demonstrating its potential to harden codebases against regressions and optimize the testing process.

6/24/2024

This post details the development and implementation of an AI-assisted root cause analysis system for incident response. The system combines heuristic-based retrieval to narrow down potential code changes and a Llama-based LLM ranker to identify the most likely root causes. The training process involved continued pre-training and supervised fine-tuning of a Llama 2 (7B) model using historical investigation data and internal documentation. The system achieved 42% accuracy in identifying root causes at the creation time of investigations related to the web monorepo. Future plans include expanding capabilities to autonomous workflow execution and proactive incident detection.

12/19/2023

This post introduces HawkEye, a toolkit for monitoring, observability, and debuggability of end-to-end machine learning workflows at Meta. HawkEye streamlines debugging by providing guided exploration, root cause analysis for AI model and feature issues, and has facilitated order of magnitude improvements in debugging time. It supports workflows for isolating top-line product issues to model snapshots, prediction anomalies to features, upstream causes of feature issues, diagnosing model snapshots, and diagnosing training data issues.

8/29/2022

This post details the evolution of Meta's SLO workflows by introducing a collaborative data annotation framework within the SLICK platform. It addresses the challenge of identifying failure patterns by enabling engineers to annotate SLO violations with structured metadata. The post describes the development of a flexible annotation schema, various methods for creating schematized annotations (CLI, UI, Workplace bot, and automated integration with Dr. Patternson), and an insights UI for visualizing and analyzing this data. This allows for better prioritization of reliability issues and a deeper understanding of root causes.

5/5/2022

Introduced BellJar, a new framework for testing system recoverability at scale. BellJar creates isolated, vacuum-sealed environments that mimic production during total infrastructure outages, allowing engineers to test recovery strategies and uncover dependencies. It uses virtualization and fault injection to simulate various failure modes and allows granular allowlisting of production capabilities needed for recovery. The framework enables programmatic definition of recovery scenarios, including the service under test, hardware, recovery strategy, validation criteria, tooling, and recovery conditions, facilitating repeatable testing and integration into CI/CD pipelines.

3/17/2022

This post introduces and details two novel approaches, Fleetscanner (out-of-production testing) and ripple (in-production testing), for detecting silent data corruptions (SDCs) at scale within Meta's infrastructure. It compares their effectiveness, highlighting that ripple testing can detect 70% of fleet data corruptions in 15 days, while opportunistic testing takes about 6 months for the same. The post emphasizes the importance of combining both methods for comprehensive SDC detection and mitigation, detailing the infrastructure trade-offs and observed metrics for each approach.

3/8/2022

Introduced a new open-source compositional static analyzer for detecting deadlocks in Android Java code. The analyzer uses abstract interpretation to compute method summaries of lock acquisition and release, identifying critical pairs that indicate potential deadlocks. It scales to large codebases by processing modified files and heuristically locating relevant methods outside the changes. The tool has been deployed in Meta's CI system, leading to over 200 deadlock fixes.

1/18/2022

This post details the migration of FOQS from a regional to a geo-distributed, global architecture to enhance disaster readiness. Key technical contributions include: 1. Architectural modifications to support routing and seamless failover of MySQL primaries across regions using global application primitives and Shard Manager. 2. Introduction of an intelligent routing service with an enqueue distributor for optimal data placement based on logical region preferences and a dequeue distributor for efficient item discovery. 3. Successful migration of tenants at scale with zero downtime for internal clients, addressing challenges like stuck queue items and high cross-region latencies inherent in the previous regional architecture.

12/16/2021

Introduced the Power Loss Siren (PLS), a rack-level, low-latency distributed power loss detection and alert system. PLS leverages existing in-rack batteries to notify services about impending power loss at least 45 seconds in advance, allowing for proactive mitigation handlers. The architecture includes PLS Relay on rack switches for detection and PLS Handler on servers for mitigation. This system improves reliability for single-primary databases by enabling proactive promotion of remote secondaries and for web servers by stopping new requests and initiating graceful shutdowns, reducing peak error rates by 100x. PLS also simplifies physical infrastructure management by reducing reliance on dual-powered rows and buffer server capacity.

12/13/2021

This post introduces SLICK, a dedicated SLO store built to standardize and centralize Service Level Indicator (SLI) and Service Level Objective (SLO) definitions across Meta. SLICK provides tooling for defining SLOs, offers high-retention, full-granularity data (up to two years) for key service metrics, and integrates SLOs into daily workflows and incident management. The architecture includes a DSL for configuration, a syncer, a UI for dashboards and index, a service for query abstraction, and data pipelines for ingestion into a sharded MySQL database. SLICK has seen widespread adoption, with over 1,000 services onboarded, and has demonstrably helped teams like LogDevice and backend ML services identify and fix reliability regressions.

10/20/2021

This post details the design and implementation of an integration testing infrastructure for backend services at Meta, along with a new autonomous testing extension. The integration testing infrastructure provides a code framework, test scheduling, and execution capabilities to encourage engineers to write effective tests. It addresses challenges in defining test environments by reusing production infrastructure (containerization, routing) to create isolated, ephemeral entities. The autonomous testing extension borrows from fuzz testing and leverages the homogeneity of the software stack to automatically generate tests. Key aspects include defining test environments with isolation layers, specifying test inputs via test harnesses and mocks, and using custom assertions or generic failure detection as test oracles. The isolation strategy for autonomous testing involves a nuanced approach combining application-level and network-level isolation to allow known read-only traffic, use allowlists for safe destinations, reroute standard RPC traffic to universal mocks, and block all other requests.

10/5/2021

This post details the technical causes and recovery process of the October 4th outage, which was triggered by a command that unintentionally took down the global backbone network. It explains the role of routers, DNS, and BGP in network connectivity, the challenges faced due to the loss of both primary and out-of-band network access, and the physical security measures that impacted recovery time. The post also discusses the 'storm' drills used to manage service restoration and the lessons learned regarding the balance between security hardening and recovery speed.

10/5/2021

This post details the root cause of a major outage on October 4th, 2021, which was attributed to configuration changes on backbone routers that disrupted inter-data center network traffic. It highlights the cascading effect of this disruption and reassures that no malicious activity or data compromise occurred.

6/2/2021

This post details Meta's approach to managing PCIe faults in its data centers, introducing a workflow and suite of tools (PCIcrawler, MachineChecker, PCIe Error Logging Service, FBAR) to detect, diagnose, remediate, and repair hardware issues. It highlights the importance of analyzing PCIe error rates (corrected and uncorrected), link speeds, and link widths, and describes automated remediation strategies such as reseating components, swapping hardware, and identifying firmware-related issues through data analysis in Scuba. The post also advocates for industry-wide adoption of PCIe AER functionality.

4/27/2021

Introduced a new reverse debugging technique for large-scale systems by integrating Intel Processor Trace (Intel PT) with eBPF for low-overhead, continuous tracing. Developed a circular buffer mechanism for efficient trace storage and an eBPF kernel probe for near-instantaneous crash notification. Enhanced LLDB with components for decoding raw traces, reconstructing instruction sequences, and building function call trees for historical analysis. Enabled latency analysis by incorporating timing information into traces and developed visualization tools for performance analysis.

4/16/2021

This post introduces De-identified Telemetry (DIT) for WhatsApp, a novel system designed to collect usage, reliability, and performance data while significantly enhancing user privacy. DIT leverages an Anonymous Credential System (ACS) and Verifiable Oblivious Pseudorandom Functions (VOPRFs) to authenticate data without the server learning its origin, thereby minimizing metadata tied to specific individuals. The post details the technical implementation, including the two-step logging workflow, curve choices, unlinkability guarantees, re-identifiability mitigation, rate limiting, and communication cost optimizations, showcasing a significant step in privacy-preserving data collection for operational insights.

2/23/2021

This post details the research and best practices for detecting and remediating silent data corruption (SDC) in large-scale infrastructure systems. It provides a real-world example of SDC in a data center application leading to missing database rows, illustrating the debug methodology and the approach to tackling it. The post highlights the complexity of detecting and reproducing SDC, the sporadic nature of failures, and the root-cause analysis involving CPU defect types and specific mathematical computations. It emphasizes the need for hardware resiliency, production detection mechanisms, and fault-tolerant software architectures to build more reliable infrastructure.

2/17/2021

Introduced the 'Fix Fast' initiative to reduce the effort required to fix regressions by improving the engineering experience and moving actionable detection further upstream. Developed a cost-per-developer (CPD) metric to incentivize 'shifting left' (early detection and fixing). Implemented strategies for earlier detection through IDE tests and predictive test selection. Improved signal quality by developing a 'meaningful action' heuristic to reduce noise and tune regression detection logic. Enhanced faster attribution by adding an 'I'm not the right owner' button to tasks and speeding up the multisect service for root cause analysis. Implemented land-blocking signals, such as @NullSafe annotations for Java files, to prevent regressions from being promoted.

2/9/2021

Introduced Minesweeper, an automated root cause analysis (RCA) system that leverages sequential pattern mining and statistical analysis (precision, recall, F1-score) on event traces to identify the causes of bugs. This system processes tens of thousands of error reports in minutes with 85% accuracy, serving as a first-line defense against bugs and reducing manual debugging efforts.

12/14/2020

This post details the development of Infer#, a new static analyzer for C# that extends Meta's Infer platform to the .NET ecosystem. It introduces a new language front end for C# by creating a compiler from .NET bytecode to Infer's intermediate language (SIL). It also adds a language-independent JSON layer for SIL, enabling future front ends for other languages. Infer# brings scalable, automated, and inter-procedural static analysis for detecting bugs like null dereferences, resource leaks, and race conditions to the .NET platform, analyzing Microsoft software such as Roslyn, DotNET SDK, ASP.NET Core, and MSBuild.

12/10/2020

Introduced the Probabilistic Flakiness Score (PFS) to quantify and monitor the reliability of automated regression tests. PFS allows for measuring the degree of flakiness for individual tests and monitoring changes in reliability over time, enabling proactive repair of unreliable tests. The system uses Bayesian inference and a statistical model to estimate flakiness based on observed test results, considering factors like code under test, state of the world, and inherent test flakiness. The model accounts for the asymmetry in how developers perceive passing and failing test results.

12/9/2020

This post introduces four key methodologies for maintaining high hardware availability in Meta's large-scale infrastructure: 1. Automated hardware remediation using MachineChecker and FBAR, with Cyborg for lower-level fixes and manual ticket creation. 2. Minimizing performance impact of error reporting by implementing a hybrid mechanism for memory error reporting using CMCI and SMI. 3. Leveraging a machine learning framework to predict and prioritize repairs for undiagnosed or misdiagnosed hardware failures. 4. Automating fleet-level root cause analysis by employing a scalable tool that uses Scuba and FP-Growth to find correlations in millions of log entries.

9/8/2020

This post details Meta's strategy for improving fault tolerance by optimizing workload placement across data centers. It introduces the concept of 'fault domains' and explains how even distribution of hardware, services, and data within these domains minimizes the impact of localized failures. The post outlines the challenges and solutions for achieving optimal placement, including the use of integer linear programming for hardware placement and continuous rebalancing for service placement, ultimately aiming to tolerate sub-data center faults without losing entire data center capacity.

8/17/2020

This post details the evolution of Meta's 'Async' system, which handles billions of asynchronous user requests. It describes the challenges of scaling the original centralized, first-come-first-served system, the subsequent development of a more robust, multitenant architecture, and key innovations like delay tolerance, optimized capacity management (queueing, time shifting, batching), and capacity regulation. This directly contributes to the site reliability by ensuring efficient and prioritized processing of critical background tasks.

3/5/2020

This post introduces Health Compass and Incident Tracker, two new systems designed to unify and streamline the process of preventing performance regressions in Meta's codebase. Health Compass provides a single configuration interface for describing performance scenarios, ensuring consistent metrics and metadata across various analysis tools. Incident Tracker leverages these scenarios to perform time-series regression detection (week-over-week and release-over-release) for beta and production analytics, consolidating relevant data to aid investigation and deduplicating issues across different environments. The systems aim to reduce distracting alerts and improve the efficiency of performance monitoring throughout the development lifecycle.

11/26/2019

Introduced CCSM (Continuous Contrast Set Mining), a novel ML-based anomaly-detection framework to accelerate the diagnosis and resolution of app crashes. CCSM extends contrast set mining techniques to continuous data, enabling engineers to identify statistically significant feature sets that differentiate crash groups from normal behavior. The algorithm is over 40 times faster than naive approaches and uses a tree search framework with ANOVA F-tests for significance and a user-defined threshold for practical significance, improving debugging efficiency for high-dimensional and mixed-type data.

11/8/2019

Introduced a Fast Dimensional Analysis (FDA) framework to automate root cause analysis on structured logs at scale. The framework combines structured logs from multiple sources, deduplicates data, and uses one-hot encoding to prepare data for frequent pattern mining. It leverages Apriori/FP-Growth algorithms to identify frequent item-sets and uses support and lift metrics for filtering and significance measurement. Pre- and post-processing, parallelism, and filters are employed to enhance scalability and interpretability.

2/8/2019

This post announces that the team behind Infer, Meta's static program analyzer, won the POPL 2019 Most Influential Paper Award for their work on "Compositional Shape Analysis by Means of Bi-abduction." It highlights Infer's use in scanning source code to detect bugs before shipping, its sophisticated interprocedural and interfile analysis at scale, and its ability to detect subtle bugs. The post also mentions the introduction of AL, a declarative language for designing new checkers, and Infer's expansion to detect concurrency bugs and apply to multithreaded rendering. Both Infer and AL have been open-sourced and are in use at other companies.

11/21/2018

Introduced a machine learning-based predictive test selection system that uses a gradient-boosted decision-tree model to probabilistically select regression tests for code changes. This system learns from historical code changes and test outcomes to estimate the likelihood of a test finding a regression, allowing it to run a smaller subset of tests (one-third of transitively dependent tests) while catching over 99.9 percent of regressions. The system also addresses test flakiness by aggressively retrying failed tests during training to distinguish true regressions from non-deterministic failures. This contributes to improved reliability and efficiency of the code development process, complementing other tools like Sapienz (automated testing) and Getafix (automated bug-fixing).

11/6/2018

Introduced Getafix, a novel tool that automatically finds and suggests fixes for bugs identified by static analysis (Infer) and automated testing (Sapienz). Getafix leverages hierarchical clustering and anti-unification to learn fix patterns from past code changes, considering both the edit and its surrounding context. It employs a tree differencer to identify granular edits and a pattern mining process that generates abstract fix patterns with 'holes' for context-specific instantiation. This enables more intelligent and context-aware bug remediation than simpler auto-fix tools.

10/9/2018

This post details the development of Safety Check for Workplace, an enterprise-grade crisis response tool derived from Facebook's public Safety Check feature. It highlights the adaptation of the core functionality (locate, notify, iterate) for business use, including enterprise-specific access management, scalability improvements using TAO for data storage, and the implementation of a robust, multi-channel notification system built on existing Messenger infrastructure. The post also emphasizes the iterative development process, starting from a hackathon idea and evolving into a production-ready product.

10/4/2018

Introduced and open-sourced StateService, a state machine as a service designed to automate the recovery of third-party services running on virtual machines (VMs) after major outages. StateService uses YAML-defined state machines to direct VM deployment processes, integrating with configuration management software like Chef to ensure sequential execution of actions and return services to their last-known state, thereby reducing manual intervention and recovery time.

7/19/2018

Introduced 'oomd', a userspace Out-Of-Memory (OOM) killer that replaces the traditional Linux kernel OOM killer. oomd utilizes Pressure Stall Information (PSI) for resource monitoring and cgroup2 for process accounting. It features pre-OOM hooks for early detection and a plugin system for custom OOM handling policies, offering faster, more flexible, and reliable OOM management compared to the kernel-based approach. This has led to the elimination of 30-minute livelocks in production environments.

5/2/2018

Introduced Sapienz, an intelligent automated software testing tool that uses search-based software testing and UI interaction modeling to design and execute test cases for the Facebook Android app. Sapienz samples the space of possible tests by building a model of the system under test through UI interactions, and harvests good tests for reuse. It employs a hybrid search approach combining individual events with higher-level motif events. Sapienz has been deployed at scale, running tens of thousands of tests daily, leading to issues being fixed within hours or minutes of code writing, with a 75 percent actionable report rate. It integrates with Facebook's One World platform for scaling emulators and reports directly into the continuous integration system, Phabricator.

10/19/2017

Introduced RacerD, an open-source static race detector for Java programs that operates without running the program. RacerD uses symbolic reasoning to identify data races, has been running in production for 10 months on Facebook's Android codebase, and has caught over 1000 multi-threading issues. The post details the challenges of concurrent programming at scale, the design decisions made for RacerD (compositionality, tracking lock/thread info, ownership analysis), and provides examples of its reasoning.

9/6/2017

This post details the development and application of the Infer static analyzer, a tool designed to find inter-procedural bugs at scale. It explains Infer's compositional, summary-based analysis technique using pre/post specifications to reason about method interactions without re-evaluating entire method bodies. The post provides concrete examples of Infer finding null dereference bugs in the DuckDuckGo Android app and OpenSSL, highlighting how its inter-procedural analysis surpasses intra-procedural tools. It also discusses the challenges of false positives in large codebases and the importance of inter-file analysis.

5/24/2017

Introduced AL, a new declarative language for easily designing static analysis checkers for Infer. AL simplifies the process of writing checkers for intraprocedural bugs by reasoning about abstract syntax trees, reducing the need for deep static analysis expertise and knowledge of Infer's internals. This allows for quicker identification and reporting of bugs, contributing to improved site reliability.

9/13/2016

This post introduces DHCPLB, an open-source load balancer developed to address significant DHCP traffic imbalance issues observed in Meta's data centers. The post details the problem of uneven request distribution across DHCP servers (KEA) due to network topology and anycast limitations, which led to outages. It describes the development of DHCPLB through hackathons and intern projects, highlighting its features like DHCPv6 support, A/B testing capabilities for KEA server updates, MAC address-based overrides, integration with monitoring infrastructure, hot-reloading configuration, BGP advertisement, and two balancing algorithms. The successful rollout of DHCPLB in production demonstrated improved load balancing and simplified testing of DHCP infrastructure, directly contributing to enhanced site reliability.

9/12/2016

This post details Meta's implementation of signed SSH certificates with principals to achieve scalable and secure access to its vast server infrastructure. It addresses the limitations of traditional password and public-key management for large-scale environments, introducing a robust system that leverages certificate authorities for authentication, principals for authorization, and centralized logging for accounting. The post also outlines the implementation details, including the creation of security domains and the configuration of SSH servers to enforce access policies.

9/9/2016

This post announces Peter O'Hearn's election as a Fellow of the UK Royal Academy of Engineering for his pioneering work on separation logic. This theory is the foundation for Facebook's open-source static analysis tool, Infer. Infer is highlighted as a system that catches over 1,000 bugs per month before code shipment, saving engineers time and improving user experience. The post details how separation logic enables Infer to model program behavior and reason about independent parts of an application, scaling to large codebases. It also mentions Infer's open-source adoption by companies like Uber and Spotify.

7/13/2016

This post details the development and scaling of a mobile device lab within the Prineville data center to support automated performance testing of the Facebook app. It describes the iterative hardware design process, from 'sled' and 'gondola' to custom-built EMI-isolated racks capable of holding 32 phones each, and the software challenges overcome using package management and Chef for consistent device configuration and monitoring. The post also outlines future plans for densifying racks, improving iOS testing automation, and extending the platform to other teams.

7/11/2016

This post introduces Aggregate Maintenance Handlers, an enhancement to the Facebook Auto Remediation (FBAR) service, designed to safely automate maintenance on multiple servers at once at the rack or multi-rack level. It details the framework for disabling and enabling servers in bulk, considering scope, maintenance type, start time, and duration to make impact-based decisions. The post also describes the four-step disable/enable process (preflight check, pre-disable, host-level disable, post-disable) and introduces Dapper as a tool to coordinate automated and manual maintenance processes, especially when automation fails or is not possible. Lessons learned from serial and parallel use of disable logic are also discussed, highlighting the importance of robust automation for large-scale infrastructure maintenance.

6/2/2016

This post details the technical evolution of Meta's Safety Check feature, focusing on streamlining deployment and improving reliability. Key contributions include the development of a push-based notification algorithm that leverages the social graph for efficient user identification in affected areas, the implementation of an internal rate-limiting service to manage job scheduling and prevent overload, and the creation of a proactive testing system with 'shadow launches' to ensure readiness. Additionally, it describes the automation of the activation process through an internal tool and the use of a Messenger bot for monitoring and reporting.

5/27/2016

This post details the extensive use of Python in Meta's production engineering for infrastructure management, operational automation, and service configuration. It highlights Python's role in managing the hardware lifecycle, network setup, auto-remediation of faults, maintenance scheduling, server imaging, and fault detection. It also covers platform services like job scheduling, blob distribution, TFTP implementation, and workflow execution. The post emphasizes Python's use in service configuration management via Configerator and Tupperware, and its role in operational efficiency through tools like MySQL Pool Scanner and slowroll orchestrator. It also notes the ongoing migration to Python 3 and the performance benefits of AsyncIO.

9/17/2015

This post details the development and deployment of "Wayfinders" (interactive digital maps) and "Room Tools" (digital calendars outside conference rooms) to improve navigation and meeting room booking at Facebook. It also describes the integration of motion sensors in conference rooms with room calendars to detect and mitigate unused bookings, leading to automated reminders and cancellations, thereby increasing room availability. The analysis of room calendar, motion sensor, and videoconferencing data provides insights into work habits and physical space utilization.

8/24/2015

This post details Meta's efforts to reduce out-of-memory (OOM) crashes in the Facebook iOS app. It describes the process of identifying OOMs through logging and process-of-elimination, profiling memory usage to pinpoint issues with UIWebView, migrating to WKWebView to offload memory usage to a separate process, optimizing allocation rates, and implementing an in-app memory profiler using Objective-C method swizzling to track object allocations and detect leaks. The post highlights the importance of addressing OOMs as a blind spot in traditional crash reporting and the impact of these efforts on app reliability.

7/20/2015

Introduced query packs for osquery, allowing users to group queries by function or problem domain for easier distribution and updates. This feature leverages osqueryd's existing scheduler to provide data differentials and alerts. The post also details the internal use of specific query packs for incident response, vulnerability management, and OS X attacks, and encourages community contributions.

6/11/2015

This post details the open-sourcing of Facebook Infer, a static program analyzer used internally to identify bugs like null pointer access and resource/memory leaks in mobile applications before shipping. It introduces the novel techniques of separation logic and bi-abduction for efficient, scalable analysis of large and rapidly evolving codebases, enabling developers to fix hundreds of potential bugs monthly.

5/15/2015

This post details discussions and technical approaches shared at the Spam Fighting @Scale conference, focusing on techniques for fighting spam and abusive content at scale. Key technical contributions include: the application of event-based processing and aggregation techniques (counter-based and cluster-based) for spam detection; the use of Haxl for simplified data fetching in Haskell; graph-based modeling for representing threat intelligence in ThreatExchange; and strategies for fighting spam in small-text fields using name and headline scoring models, and account clustering. The post also emphasizes the importance of product changes over classifiers and understanding the economics of spam.

4/10/2015

This post introduces CT-Scan, a performance monitoring and prediction platform designed to detect and prevent performance regressions in Facebook's mobile applications. It details the system's design principles, focusing on pattern detection and diagnostics across development, staging, and production environments. The post outlines the implementation of A/B test-style experiments in development for rapid feedback, continuous experiments and triaging runs in staging for comprehensive analysis, and sampled real-world perf counter collection in production. It also discusses the challenges of building a scalable and extensible system, including the development of a device lab and a zookeeper-based scheduling service.

2/18/2015

Introduced Stetho, an open-source debugging platform for Android that integrates with Chrome Developer Tools. Stetho allows developers to inspect network activity and application data, and extends Android's `dumpsys` functionality with custom plugins for more convenient debugging and internal settings manipulation.

11/5/2014

This post details the "Security @Scale 2014" conference, highlighting Meta's and other industry leaders' efforts to build scalable security solutions. It showcases advancements in bug bounty programs (HackerOne), vulnerability management (Square's Report Card, Doorman, Sting), scalable rule engines for spam fighting (Facebook's Haxl), widespread SSL adoption (Twitter), host intrusion detection (Facebook's osquery), and incident response strategies (Facebook's homebrew IR and Network Security Monitoring, GitHub's ChatOps). This contributes to the understanding of how Meta addresses large-scale security challenges and collaborates with the industry to improve overall online safety.

10/29/2014

Introduced osquery, a framework that exposes operating systems as a high-performance relational database, allowing SQL-based queries for low-level OS monitoring. This includes an interactive console (osqueryi) for exploration and a daemon (osqueryd) for large-scale host monitoring, with cross-platform support and a modular codebase for extensibility. The post details example queries for process monitoring and network connections, and announces the open-sourcing of osquery.

8/12/2014

This post details the debugging process for a file corruption issue on iOS affecting the Facebook app. It describes the use of Hipal and Scuba for crash report analysis, the hypothesis-driven approach to isolate the cause (switching from Core Data to SQLite), the 'honeypot' file technique to detect rogue writes, the use of lldb for conditional breakpoints on the `write` system call, and the application of Fishhook to rebind system APIs for direct proof of the networking stack writing to a closed socket that was reassigned to the database file. The root cause was identified as a file descriptor lifetime handling issue in the SSL layer of the SPDY network stack.

11/12/2013

This post introduces the concept of 'mature optimization' as a disciplined approach to performance tuning, emphasizing measurement, cost-benefit analysis, and deferring optimization until systems are mature. It cautions against premature optimization and highlights that significant gains are often concentrated in a small portion of code, discoverable through careful measurement. The post also touches upon performance as a feature when resources are limited or post-ship changes are difficult.

7/11/2013

This post introduces Windex, an automated system for database provisioning, which complements existing automation for detecting and resolving MySQL host issues (MPS). Windex handles the re-imaging, OS installation, and MySQL configuration of hosts, ensuring a pool of spare MySQL server instances is always available. It includes robust validation, retry logic, and failure pattern detection to prevent damage and identify root causes of provisioning failures, thereby enhancing the overall site reliability and efficiency of database infrastructure management.

1/24/2013

Introduced FXL (Feature eXtraction Language), a domain-specific language optimized for spam fighting at Facebook. FXL is a subset of Standard ML designed for efficient data fetching across the graph. Its core innovation is the use of pure functions, which enables automatic optimizations like batching of data requests (leading to a ~20x speedup) and memoization of common expressions, significantly improving performance and enabling rapid adaptation to new spam threats.

4/5/2012

This post details the evolution of Facebook's release engineering process, emphasizing the 'daily push' culture and the development of tools and processes to support rapid iteration and high-frequency code deployments. It introduces the concept of 'push karma' as a mechanism to manage risk associated with code changes and highlights the importance of test automation and operational vigilance in maintaining site reliability. The post also touches upon the engineer's responsibility for their code from submission to production.

4/3/2012

This post introduces the work of the Site Performance team, which focuses on automatically finding problems with site speed and reliability. They collect page load information from servers and use machine learning to identify trends, pinpoint issues, and trace them back to the responsible engineer. This contributes to the ongoing effort of maintaining high site reliability despite rapid code shipping.

9/15/2011

Introduced Facebook Auto-Remediation (FBAR), an automated system for detecting and remediating server and software outages. FBAR utilizes a monitoring system to detect alerts, processes them to determine appropriate workflows, and executes remediation plugins against a generic API modeling the infrastructure. This system handles tasks like removing faulty servers from production and flagging them for repair, significantly reducing manual intervention and increasing service reliability.

9/6/2011

This post details the evolution of Facebook's site reliability efforts, focusing on the development of a comprehensive error logging and monitoring system. It describes the implementation of server-side and client-side error logging using Scribe, the processing of massive error data with Apache Hive for trend analysis, and the calculation of reliability metrics. The post highlights the identification and fixing of various user-visible errors, including issues in News Feed comments, Chat presence servers, database access, and back-end services. It also outlines the creation of a real-time monitoring system with over 1500 trend lines and the use of machine learning for anomaly detection and alarm generation, which has resulted in a 5x reduction in user-visible errors.

5/12/2011

This post introduces 'Login Approvals,' a new opt-in security feature that implements two-factor authentication using SMS codes sent to a user's mobile phone. It highlights the engineering effort, including intern contributions, to balance security with usability, opting for SMS over app-based or hardware tokens for broader accessibility. The feature aims to protect user accounts from unauthorized access by requiring verification from new or unrecognized devices, with options to save trusted devices and recover access if a phone is lost.

9/24/2010

This post details a severe outage caused by an automated configuration verification system that incorrectly handled an invalid configuration change. The system's attempt to correct invalid cache values by querying a persistent store led to a feedback loop when the persistent store itself contained an invalid value. This resulted in an overwhelming number of queries to a database cluster, causing it to fail. The only way to recover was to shut down the entire site. The post highlights the need for configuration systems to be designed to gracefully handle feedback loops and transient spikes, and to avoid cascading failures when dealing with invalid persistent data.

2/8/2010

This post introduces the concept and structure of Site Reliability Engineering (SRE) at Facebook, emphasizing the team's responsibility for ensuring the site's uptime, performance, and feature reliability. It details the 'Site', 'Reliability', and 'Engineering' aspects of SRE work, highlighting the development of automated tools for server management, the handling of complex infrastructure like large memcached footprints, and the coordination with Release Engineering. The post also quantifies the SRE team's impact with a high user-to-SRE ratio and discusses the dynamic and challenging nature of the work.

7/2/2009

This post details the operational engineering challenges and solutions for launching the username feature to 200 million users simultaneously. It describes the use of Memcached infrastructure for name availability checks and blocked names, a stripped-down registration page to minimize load, strategic launch timing, and the establishment of a 'War Room' for real-time monitoring and response. Key technical strategies included 'dark launching' to pre-test backend functionality under simulated load and the implementation of 'levers' and 'nuclear options' for dynamic throttling of site features (e.g., Facebook Chat, story counts, commenting/liking) to manage unexpected traffic surges.

4/3/2009

Introduced XHProfLive, a system for continuous, function-level performance statistics from production. XHProfLive leverages XHProf, an instrumentation-based callgraph profiler for PHP, to gather detailed metrics like execution times and memory usage, broken down by callers and callees. It provides a rich UI for trend tracking, comparison reports, and automated regression alerts, significantly aiding performance diagnosis and optimization.