TechPrufer
Get started
Meta
Meta logoMeta · Feature trail

Site Reliability and Outage Management

70 posts · Apr 2009 – Jun 2026

What this trail covers

Meta's site reliability and outage management has evolved to include automated root cause analysis (RCA) for hardware and system failures using techniques like MachineChecker, FBAR, Cyborg, and a scalable RCA tool that sorts through millions of log entries using pre-aggregation with Scuba and the FP-Growth algorithm. The system analyzes chronological traces of user events to identify statistically distinctive patterns correlated with bugs, enabling faster and more accurate debugging. Oomd, a use case for AI-assisted root cause analysis, leverages heuristic-based retrieval and LLM-based ranking to identify potential code changes responsible for incidents, achieving 42% accuracy in identifying root causes at the creation time of investigations related to their web monorepo. This involves fine-tuning Llama 2 models with historical investigation data and employing strategies like ranking through election to manage context window limitations. Future work includes expanding capabilities to autonomous workflow execution and proactive incident detection.

All Meta trails
  1. Lights Out, Systems On: Validating Instant Power Loss ReadinessThis post introduces 'Instantaneous PowerLoss Storm,' a new testing paradigm for validating zero-notice power loss readiness in Meta's data center regions. It details the defense-in-depth strategies implemented across the DC stack, including battery backup, Power Loss Siren (PLS), and unavailability events (UEs) for the Twine orchestrator. Key challenges addressed include bootstrapping an entire region, specifically circular dependencies among control plane services (solved with Belljar tests and a Twine recovery kit) and the 'boomerang' effect where UEs impacted orchestrators (mitigated by allowing control plane services to ignore power-related UEs). Tradeoffs were defined, distinguishing between critical impacts (data loss, facility damage) and tolerable risks (transient errors, bounded staleness). Validation is incremental, progressing from pre-production to production regions, with the goal of handling region loss as seamlessly as sub-regional fault domains.
  2. DrP: Meta’s Root Cause Analysis Platform at ScaleThis post introduces DrP, Meta's Root Cause Analysis (RCA) platform, which automates incident investigation. DrP consists of an expressive SDK for authoring 'analyzers' (codified investigation workflows), a scalable backend for executing these analyzers, integration with alerting and incident management tools for auto-triggering, and a post-processing system for automated mitigation actions. Key technical components include ML algorithms for anomaly detection, event isolation, time series correlation, and dimension analysis within the SDK. The backend supports multi-tenant and isolated execution environments. DrP has demonstrated significant MTTR reduction (20-80%) and is adopted by over 300 teams, executing 50,000 analyses daily. The platform is evolving towards an AI-native approach for enhanced automated investigations.
  3. Zoomer: Powering AI Performance at Meta’s Scale Through Intelligent Debugging and OptimizationIntroduced Zoomer, an automated debugging and optimization platform for Meta's AI training and inference workloads. Zoomer provides deep performance insights to enable energy savings, workflow acceleration, and efficiency gains. Its architecture comprises three layers: an infrastructure layer for scalable trace data storage and processing (using Manifold), an analytics and insights engine with specialized analyzers (integrating Kineto, NVIDIA DCGM, StrobeLight, dyno telemetry) for GPU, CPU, communication, and memory analysis, and a visualization layer for interactive timelines, dashboards, and automated summaries. Zoomer automates profiling triggers, captures comprehensive data streams (GPU metrics, execution traces, host-level data, application annotations, inference-specific data, communication analysis), and performs distributed analysis including straggler detection, bottleneck analysis, critical path analysis, anti-pattern detection, parallelism analysis, memory analysis, and load imbalance analysis. It offers specialized features for LLM workloads (N-dimensional parallelism visualization, GPU efficiency heat maps) and inference optimization (single-click QPS optimization, request-level deep dives, real-time memory profiling).
  4. Diff Risk Score: AI-driven risk-aware software developmentThis post introduces Diff Risk Score (DRS), an AI-powered technology built on a fine-tuned Llama LLM that predicts the likelihood of code changes causing production incidents (SEVs). It highlights DRS's impact on enabling code unfreezes during sensitive periods, leading to significant productivity gains and reduced engineering time spent on incident mitigation. The post also outlines future directions for risk awareness in software development, including expanding to configuration change risk, automating risk mitigation, and improving natural language explanations for risk-aware technologies.
  5. How Meta keeps its AI hardware reliableThis post details Meta's methodologies for detecting Silent Data Corruptions (SDCs) in AI hardware, a critical challenge for reliable AI training and inference. It categorizes hardware faults into static, transient, and silent errors, highlighting the unique difficulties of SDCs due to their data dependency and subtle manifestation. The post introduces and elaborates on three novel SDC detection mechanisms: Fleetscanner (targeted micro-benchmarks), Ripple (co-located workload tests), and Hardware Sentinel (application exception analysis). It further discusses the specific implications of SDCs in AI training workloads, including NaN propagation and corrupted gradient variance, and their impact on inference workloads, leading to incorrect results. The contribution is the detailed explanation of these SDC detection and mitigation strategies specifically for AI hardware, building upon existing site reliability efforts.
  7. Revolutionizing software testing: Introducing LLM-powered bug catchersThis post introduces the Automated Compliance Hardening (ACH) tool, a novel system that leverages Large Language Models (LLMs) for mutation-guided test generation. ACH automatically generates realistic faults (mutants) in source code and then creates tests specifically designed to catch these faults. This approach moves beyond traditional code coverage-focused testing by targeting specific bugs, with verifiable assurances that the generated tests will indeed catch the intended faults. ACH has been applied to critical platforms like Facebook Feed, Instagram, Messenger, and WhatsApp, demonstrating its potential to harden codebases against regressions and optimize the testing process.
  8. Strobelight: A profiling service built on open source technologyIntroduced Strobelight, Meta's profiling orchestrator, which combines multiple profilers (many open source, often using eBPF) to collect detailed performance metrics (CPU usage, memory allocations, etc.) from all production hosts. Strobelight enables engineers to identify performance bottlenecks, optimize code, and improve utilization, leading to significant efficiency wins (e.g., 15,000 servers' worth of annual capacity savings). It supports 42 different profilers, including memory, function call count, event-based, AI/GPU, off-CPU time, and request latency profilers. Strobelight allows for on-demand profiling via CLI/UI and continuous/triggered profiling via configuration files. It includes safeguards against performance degradation and profiler conflicts. A core principle is providing automatic, regularly-collected profiling data for all services, with dynamic sampling and weighted samples for aggregation. The LBR profiler feeds into FDO pipelines for binary optimization, and the event profiler collects stack traces for monitoring and testing. Stack Schemas enhance data with tags for better analysis.
  9. How Meta built large-scale cryptographic monitoringThis post details the implementation of a large-scale cryptographic monitoring system within Meta's FBCrypto library. Key technical contributions include a buffering and flushing strategy to aggregate cryptographic events and reduce write throughput, partially randomized flushing to normalize write cadence, and aggregation of derived crypto operations to reduce logging volume. The use of Folly's ConcurrentHashMap for efficient multithreaded buffering is also highlighted. The system aims to provide deep visibility into cryptography usage for proactive security vulnerability mitigation and infrastructure reliability.
  10. Leveraging AI for efficient incident responseThis post details the development of an AI-assisted root cause analysis system for incident response. The system combines a heuristic-based retriever to reduce the search space of potential code changes and a Llama 2 LLM-based ranker to identify the top five most likely root causes. The LLM was fine-tuned using historical investigation data and a dedicated RCA SFT dataset. The system achieved 42% accuracy in identifying root causes at the investigation's creation time for the web monorepo. Future plans include autonomous workflow execution and proactive incident detection.
  11. Logarithm: A logging engine for AI training workflows and servicesThis post introduces Logarithm, a new internal logging engine designed for high-throughput ingestion (100+ GB/s) and interactive querying (thousands of queries/sec) of system and AI training logs. It details Logarithm's data model, its application in debugging AI training workflows by attaching metadata like rank IDs to log lines, and its system architecture. Key technical contributions include a tiered storage approach using Bloom filters for lightweight secondary indices, separation of compute and storage, and separation of ingestion and query clusters to ensure performance guarantees. The system is written in C++20 and supports typed structures and regex-based extraction.
  12. AI debugging at Meta with HawkEyeThis post introduces HawkEye, a toolkit for monitoring, observability, and debugging of end-to-end machine learning workflows at Meta. It details workflows for isolating top-line product issues to specific model snapshots, correlating degradation with model snapshots and feature importance, tracing feature issues upstream to data pipelines, diagnosing model snapshot issues by comparing weights and biases, and identifying training data problems. HawkEye aims to significantly reduce debugging time and simplify operational workflows for AI/ML systems.
  13. Improving Meta’s SLO workflows with data annotationsThis post introduces enhancements to Meta's SLICK SLO management platform by integrating a collaborative data annotation framework. It details the challenges of identifying failure patterns from unstructured SLO violation data and proposes a solution involving structured annotations. The post describes the development of a flexible annotation schema, various methods for creating schematized annotations (CLI, UI, Workplace bot, and automated integration with Dr. Patternson), and an insights UI for visualizing aggregated annotation data. The key technical contributions include the design of the `annotation_config` field for defining hierarchical matchers, the implementation of interactive annotation creation tools, and the development of an insights dashboard for analyzing root causes and metadata distributions.
  14. BellJar: A new framework for testing system recoverability at scaleIntroduced BellJar, a new framework for testing system recoverability at scale. BellJar provides an isolated, vacuum-sealed experimentation environment that simulates total infrastructure outages by default, allowing teams to test recovery strategies and identify dependencies. It uses an additive allowlisting scheme to selectively reintroduce necessary production capabilities, enabling granular testing of system startup procedures and validation criteria. BellJar aims to replace tribal knowledge and documentation with codified, repeatable recovery tests deployable in CI/CD pipelines.
  15. Detecting silent errors in the wild: Combining two novel approaches to quickly detect silent data corruptions at scaleThis post introduces and evaluates two novel approaches for detecting silent data corruptions (SDCs) at scale: opportunistic testing (Fleetscanner) and in-production testing (ripple testing). Opportunistic testing leverages planned maintenance events for longer, more intrusive tests, while ripple testing integrates brief, performance-aware tests alongside production workloads for faster detection of certain error types. The post demonstrates that combining both methods provides comprehensive coverage, with ripple testing detecting 70% of fleet corruptions in 15 days, and opportunistic testing covering the remaining 23% over 6 months. It highlights the trade-offs and benefits of each approach, emphasizing the need for both to ensure fleet resiliency and prevent data loss.
  16. An open source compositional deadlock detector for Android JavaIntroduced a new open-source compositional static analyzer for detecting deadlocks in Android Java code. The analyzer uses abstract interpretation to compute method summaries of lock acquisition and release, identifying critical pairs that indicate potential deadlocks. It scales to large codebases by processing modified files and heuristically locating relevant methods outside the changes. The tool has been deployed in Meta's CI system, leading to over 200 deadlock fixes.
  17. FOQS: Making a distributed priority queue disaster-readyThis post details the migration of FOQS from a regional to a geo-distributed, global architecture to enhance disaster readiness. Key technical contributions include: 1. Architectural modifications to support routing and seamless failover of MySQL primaries across regions using global application primitives and Shard Manager. 2. Introduction of an intelligent routing service with an enqueue distributor for optimal data placement based on logical region preferences and a dequeue distributor for efficient item discovery. 3. Successful migration of tenants at scale with zero downtime for internal clients, addressing challenges like stuck queue items and high cross-region latencies inherent in the previous regional architecture.
  18. Power Loss Siren: Making Meta resilient to power loss eventsIntroduced the Power Loss Siren (PLS), a rack-level, low-latency, distributed system for detecting and alerting on impending power loss events. PLS leverages existing in-rack batteries to provide advance notification (45 seconds) to services, enabling proactive mitigation handlers to failover or gracefully shut down before server power is lost. The architecture includes PLS Relay on rack switches for detection and PLS Handler on servers for mitigation. PLS has demonstrated significant improvements in service reliability, reducing peak error rates during power loss events by 100x for web servers and enabling near-invisible failover for single-primary databases. It also simplifies physical infrastructure management by reducing the need for dual-powered rows and associated buffer capacity.
  19. SLICK: Adopting SLOs for improved reliabilityThis post introduces SLICK, a dedicated SLO store built to centralize SLI and SLO definitions at Meta. SLICK provides tooling support for defining SLOs, offers high-retention, full-granularity data (per-minute, up to two years) for key service metrics, and integrates SLOs into various workflows (on-call tooling, incident declaration criteria). It aims to improve discoverability of service reliability metrics, enable long-term trend analysis, and standardize reliability expectations across the company. The architecture includes a DSL for defining SLOs, a syncer, UI for dashboards and index, a service for querying SLOs, and data pipelines for ingesting SLI time series data into a sharded MySQL database. Over 1,000 services were onboarded by 2021, with examples of its use in improving LogDevice and a back-end ML service reliability.
  20. Autonomous testing of services at scaleThis post details the design and implementation of an autonomous testing extension built on top of Meta's existing integration testing infrastructure for backend services. Key contributions include: 1. Reusing production infrastructure (containerization, routing) to build ephemeral, isolated test environments. 2. Implementing a nuanced isolation strategy that allows known read-only traffic, service owner-defined allowlists, rerouting to universal mocks, and blocking all other network requests. 3. Utilizing application-level (API-based) and network-level (IP:port-based) isolation, with stack trace inspection for code-level decisions. 4. Employing LD_PRELOAD over BPF for the isolation layer implementation. 5. Demonstrating how this system enables autonomous testing for one-third of Meta's services.
  21. More details about the October 4 outageThis post details the technical causes and recovery process of the October 4th outage, which was triggered by a command that unintentionally took down the global backbone network. It explains the role of routers, DNS, and BGP in network connectivity, the challenges faced due to the loss of both primary and out-of-band network access, and the physical security measures that impacted recovery time. The post also discusses the 'storm' drills used to manage service restoration and the lessons learned regarding the balance between security hardening and recovery speed.
  22. Update about the October 4th outageThis post details the root cause of a major outage on October 4th, 2021, which was attributed to configuration changes on backbone routers that disrupted inter-data center network traffic. It highlights the cascading effect of this disruption and reassures that no malicious activity or data compromise occurred.
  23. How Facebook deals with PCIe faults to keep our data centers running reliablyThis post details the methodology and tools developed to address PCIe faults in Meta's data centers. It introduces PCIcrawler for topology and error reporting, MachineChecker for hardware evaluation, and an in-house service for parsing dmesg/SELs to monitor PCIe error rates. The post explains how both uncorrected and corrected PCIe errors are handled, with a focus on establishing thresholds for corrected error rates that trigger hardware replacement. It also covers the detection and remediation of bad link speeds and widths, often resolved by reseating components. The workflow includes periodic checks, upstream/downstream analysis, and integration with configuration management systems. The system aims to automatically detect, diagnose, remediate, and repair PCIe faults, contributing to hardware health coverage and the reliable operation of millions of servers.
  24. Reverse debugging at scaleIntroduced a new reverse debugging technique leveraging Intel Processor Trace (Intel PT) and eBPF for efficient production-level tracing and crash analysis. Developed a circular buffer mechanism for continuous trace collection with minimal memory overhead. Utilized eBPF kernel probes for low-latency crash detection and trace buffer copying. Integrated with LLDB to decode raw traces, reconstruct instruction history, and build function call trees for human-readable analysis. Enabled latency analysis by incorporating timing information into traces. Planned integration with VSCode for a visual reverse debugging experience.
  25. DIT — enabling de-identified data collection on WhatsAppThis post introduces De-identified Telemetry (DIT) for WhatsApp, a novel system designed to collect usage, reliability, and performance data while significantly enhancing user privacy. DIT leverages an Anonymous Credential System (ACS) and Verifiable Oblivious Pseudorandom Functions (VOPRFs) to authenticate data without the server learning its origin, thereby minimizing metadata tied to specific individuals. The post details the technical implementation, including the two-step logging workflow, curve choices, unlinkability guarantees, re-identifiability mitigation, rate limiting, and communication cost optimizations, showcasing a significant step in privacy-preserving data collection for operational insights.
  26. Mitigating the effects of silent data corruption at scaleThis post details the research and best practices for detecting and remediating silent data corruption (SDC) in large-scale infrastructure systems. It provides a real-world example of SDC in a data center application leading to missing database rows, illustrating the debug methodology and the approach to tackling it. The post highlights the complexity of detecting and reproducing SDC, the sporadic nature of failures, and the root-cause analysis involving CPU defect types and specific mathematical computations. It emphasizes the need for hardware resiliency, production detection mechanisms, and fault-tolerant software architectures to build more reliable infrastructure.
  27. Faster, more efficient systems for finding and fixing regressionsThis post details the 'Fix Fast' initiative aimed at reducing the effort and time required to fix regressions in production. It introduces the Cost Per Developer (CPD) metric to incentivize 'shifting left' (earlier detection). Key improvements include: predictive test selection for CI, IDE-integrated tests for faster feedback, signal quality improvements via 'meaningful action' heuristic and noise reduction (e.g., Health Compass), faster attribution using services like Multisect and an 'I'm not the right owner' button, and land-blocking signals like NullSafe annotations for Java to prevent NPE crashes. The post highlights a data-driven approach to measuring and improving the regression fixing process.
  28. Minesweeper automates root cause analysis as a first-line defense against bugsThis post introduces Minesweeper, a new technique for automating root cause analysis (RCA) of bugs. It leverages sequential pattern mining on user event traces, comparing traces from users who encountered a bug (test group) with those who did not (control group). Statistical analysis of event patterns, using precision, recall, and F1-score, identifies the most distinctive patterns correlated with bugs. The system uses the PrefixSpan algorithm for efficient pattern mining and addresses usability challenges by avoiding redundant patterns. Minesweeper has demonstrated 85% accuracy in identifying root causes for tens of thousands of reports in minutes, significantly improving regression analysis and outage mitigation.
  29. Infer powering Microsoft’s Infer#, a new static analyzer for C#This post details the development of Infer#, a new static analyzer for C# that extends Meta's Infer platform to the .NET ecosystem. It introduces a new language front end for C# by creating a compiler from .NET bytecode to Infer's intermediate language (SIL). It also adds a language-independent JSON layer for SIL, enabling future front ends for other languages. Infer# brings scalable, automated, and inter-procedural static analysis for detecting bugs like null dereferences, resource leaks, and race conditions to the .NET platform, analyzing Microsoft software such as Roslyn, DotNET SDK, ASP.NET Core, and MSBuild.
  30. How do you test your tests?Introduced the Probabilistic Flakiness Score (PFS) to quantify and monitor the reliability of automated regression tests. PFS allows for measuring the degree of flakiness for individual tests and monitoring changes in reliability over time, enabling proactive repair of unreliable tests. The system uses Bayesian inference and a statistical model to estimate flakiness based on observed test results, considering factors like code under test, state of the world, and inherent test flakiness. The model accounts for the asymmetry in how developers perceive passing and failing test results.
  31. How Facebook keeps its large-scale infrastructure hardware up and runningThis post introduces four methodologies for maintaining hardware availability: 1. Automated hardware remediation using MachineChecker, FBAR, and Cyborg. 2. Minimizing performance impact of error reporting by using a hybrid CMCI/SMI mechanism for memory errors. 3. Leveraging ML to predict and prioritize repairs for undiagnosed/misdiagnosed tickets. 4. Automating fleet-level root cause analysis using Scuba and FP-Growth for log analysis.
  32. Introducing resctl-demo: Better resource control with simulationIntroduced resctl-demo, a tool for simulating system resource conflicts and testing resolution strategies. This tool provides developers with a guided tour of resource control, including live demos and explanations, to build intuitive knowledge for improving server reliability, responsiveness, and utilization. It offers an AMI for AWS and a USB drive image for local installation.
  33. Fault tolerance through optimal workload placementThis post details Meta's strategy for improving fault tolerance by optimizing workload placement across data centers. It introduces the concept of 'fault domains' and explains how even distribution of hardware, services, and data within these domains minimizes the impact of localized failures. The post outlines the challenges and solutions for achieving optimal placement, including the use of integer linear programming for hardware placement and continuous rebalancing for service placement, ultimately aiming to tolerate sub-data center faults without losing entire data center capacity.
  34. Asynchronous computing @Facebook: Driving efficiency and developer productivity at Facebook scaleThis post details the evolution of Meta's 'Async' system, which handles billions of asynchronous user requests. It describes the challenges of scaling the original centralized, first-come-first-served system, the subsequent development of a more robust, multitenant architecture, and key innovations like delay tolerance, optimized capacity management (queueing, time shifting, batching), and capacity regulation. This directly contributes to the site reliability by ensuring efficient and prioritized processing of critical background tasks.
  35. Preventing performance regressions with Health Compass and Incident TrackerIntroduced Health Compass and Incident Tracker to unify performance regression detection across the development lifecycle. Health Compass provides a single configuration system for performance scenarios, ensuring consistent metrics and metadata across various tools. Incident Tracker applies time series analysis (week-over-week and release-over-release) to detect regressions in production and beta environments, deduplicating issues and providing a consolidated view for investigation. This approach significantly reduces distracting alerts and improves the efficiency of performance regression management.
  36. CCSM: Scalable statistical anomaly detection to resolve app crashes fasterIntroduced Continuous Contrast Set Mining (CCSM) as a novel anomaly-detection framework to address the challenges of diagnosing and resolving mobile app crashes. CCSM extends standard contrast set mining (CSM) from categorical data to continuous data by adapting tree search algorithms and multiple hypothesis testing. It uses ANOVA F-tests for significance and a user-defined threshold for largeness in continuous data, and chi-squared tests for categorical data. The algorithm is over 40 times faster than naive baselines and provides human-readable insights by identifying statistically interesting feature sets that differentiate crash groups from control groups. The system also outputs confidence intervals and an anomaly score based on Cohen's d for improved actionability.
  37. Fast dimensional analysis for root cause analysis at scaleIntroduced a Fast Dimensional Analysis (FDA) framework to automate root cause analysis on structured logs at scale. The framework combines logs from multiple sources, deduplicates data at query time, one-hot encodes features, and applies frequent pattern mining (Apriori/FP-Growth) with support and lift metrics to identify correlated features that explain failure events. Pre- and post-processing, parallelism, and filters were implemented to further speed up analysis and improve interpretability.
  38. Systems @Scale 2019 New York recapThis post is a recap of the Systems @Scale 2019 New York event, focusing on observability. It highlights talks on "deep systems" and the breakdown of traditional observability, the use of performance analysis tools (CV and Tracery) at Facebook, Uber's approach to comprehending microservice architectures with distributed tracing, aggregate analysis of distributed traces beyond performance troubleshooting, Instagram's web server profiling framework, Google's Monarch monitoring infrastructure, Etsy's broader view of observability beyond production, Facebook's Scribe monitoring system, Datadog's scaling of observability data, Facebook's Scuba platform for real-time log analytics, and Squarespace's approach to developing meaningful SLIs. The post itself does not introduce new technical capabilities but summarizes advancements and discussions in the field of observability.
  39. Team behind Infer wins POPL 2019 Most Influential Paper AwardThis post announces that the team behind Infer, Meta's static program analyzer, won the POPL 2019 Most Influential Paper Award for their work on "Compositional Shape Analysis by Means of Bi-abduction." It highlights Infer's use in scanning source code to detect bugs before shipping, its sophisticated interprocedural and interfile analysis at scale, and its ability to detect subtle bugs. The post also mentions the introduction of AL, a declarative language for designing new checkers, and Infer's expansion to detect concurrency bugs and apply to multithreaded rendering. Both Infer and AL have been open-sourced and are in use at other companies.
  40. Predictive test selection: A more efficient way to ensure reliability of code changesIntroduced a predictive test selection system that uses machine learning (gradient-boosted decision trees) to probabilistically select regression tests for code changes. This system learns from historical code changes and test outcomes to estimate the likelihood of a test detecting a regression, allowing it to run a smaller subset of tests (one-third of transitively dependent tests) while catching over 99.9% of regressions. It also addresses test flakiness by aggressively retrying failed tests during training data collection.
  41. Getafix: How Facebook tools learn to fix bugs automaticallyIntroduced Getafix, a novel tool that automatically finds and suggests fixes for bugs identified by static analysis (Infer) and automated testing (Sapienz). Getafix leverages hierarchical clustering and anti-unification to learn fix patterns from past code changes, considering both the edit and its surrounding context. It employs a tree differencer to identify granular edits and a pattern mining process that generates abstract fix patterns with 'holes' for context-specific instantiation. This enables more intelligent and context-aware bug remediation than simpler auto-fix tools.
  42. Facebook releases Safety Check crisis response tool for WorkplaceThis post details the development of Safety Check for Workplace, an enterprise-grade crisis response tool derived from Facebook's public Safety Check feature. It highlights the adaptation of the core functionality (locate, notify, iterate) for business use, including enterprise-specific access management, scalability improvements using TAO for data storage, and the implementation of a robust, multi-channel notification system built on existing Messenger infrastructure. The post also emphasizes the iterative development process, starting from a hackathon idea and evolving into a production-ready product.
  43. Open-sourcing StateService: Automating recovery of third-party services after a major outageThis post introduces and open-sources StateService, a state machine as a service designed to automate the recovery of third-party services after major outages. StateService directs the state of VMs through complex deployment processes, reducing manual effort and recovery time. It integrates with configuration management software like Chef, using YAML to describe state transitions and HTTP for communication. This allows for programmatic control over deployment steps, ensuring they occur in the correct sequence and on specific VMs.
  44. Open-sourcing oomd, a new approach to handling OOMsThis post introduces oomd, a new userspace Out-Of-Memory (OOM) killer designed to be a more reliable and flexible replacement for the traditional Linux kernel OOM killer. Oomd leverages Pressure Stall Information (PSI) and cgroup2 to monitor system resource usage and proactively take corrective actions before a system-wide OOM occurs. It features pre-OOM hooks for early visibility and a custom plugin system to allow workload-specific handling policies, including a configurable 'kill list' for prioritizing process termination based on memory pressure and consumption. The post details synthetic test results demonstrating oomd's effectiveness in preventing livelocks and degraded performance in various OOM scenarios, and announces its open-sourcing.
  45. Sapienz: Intelligent automated software testing at scaleIntroduced Sapienz, an intelligent automated software testing tool that uses search-based software testing and UI interaction modeling to design and execute test cases for the Facebook Android app. Sapienz samples the space of possible tests by building a model of the system under test through UI interactions, and harvests good tests for reuse. It employs a hybrid search approach combining individual events with higher-level motif events. Sapienz has been deployed at scale, running tens of thousands of tests daily, leading to issues being fixed within hours or minutes of code writing, with a 75 percent actionable report rate. It integrates with Facebook's One World platform for scaling emulators and reports directly into the continuous integration system, Phabricator.
  46. Open-sourcing RacerD: Fast static race detection at scaleIntroduced RacerD, an open-source static race detector for Java programs that operates without running the program. RacerD uses symbolic reasoning to identify data races, has been running in production for 10 months on Facebook's Android codebase, and has caught over 1000 multi-threading issues. The post details the challenges of concurrent programming at scale, the design decisions made for RacerD (compositionality, tracking lock/thread info, ownership analysis), and provides examples of its reasoning.
  47. Finding inter-procedural bugs at scale with Infer static analyzerThis post details the development and application of the Infer static analyzer, a tool designed to find inter-procedural bugs at scale. It explains Infer's compositional, summary-based analysis technique using pre/post specifications to reason about method interactions without re-evaluating entire method bodies. The post provides concrete examples of Infer finding null dereference bugs in the DuckDuckGo Android app and OpenSSL, highlighting how its inter-procedural analysis surpasses intra-procedural tools. It also discusses the challenges of false positives in large codebases and the importance of inter-file analysis.
  48. AL: A new declarative language for detecting bugs with InferIntroduced AL, a new declarative language for easily designing static analysis checkers for Infer. AL simplifies the process of writing checkers for intraprocedural bugs by reasoning about abstract syntax trees, reducing the need for deep static analysis expertise and knowledge of Infer's internals. This allows for quicker identification and reporting of bugs, contributing to improved site reliability.
  49. Hardware Analytics and Lifecycle Optimization (HALO) at FacebookThis post introduces the Hardware Analytics and Lifecycle Optimization (HALO) system, which enhances site reliability by providing a comprehensive approach to monitoring, processing, and visualizing hardware data at scale. HALO integrates data from Lumberjack, asset/ticketing systems, and ODS to track SMART attributes, component logs, and environmental sensors. It employs anomaly detection, the Attribot tool for known issue identification, and Systemic Issue Detection (SID) for identifying emerging hardware problems. The outcomes include improved global hardware health through vendor collaboration and supply chain refinement, and informed hardware redesigns based on field experience, such as improved toolless serviceability and airflow design.
  50. Continuous MySQL backup validation: Restoring backupsThis post details the ORC (ORC Restore Coordinator) system for continuous MySQL backup validation. It introduces the architecture of ORC, including the Warchief (load balancer), ORC DB (state management), and Peons (restore workers). It elaborates on the five stages of a restore job within a peon: SELECT, DOWNLOAD, LOAD, VERIFY, and REPLAY. A significant technical contribution is the detailed explanation of binlog selection logic using GTIDs and `gtid_executed` values for precise transaction replay, enabling point-in-time restores. The post also outlines the challenges overcome in the restore process for a single database.
  51. DHCPLB: An open source load balancerThis post introduces DHCPLB, an open-source load balancer developed to address significant DHCP traffic imbalance issues observed in Meta's data centers. The post details the problem of uneven request distribution across DHCP servers (KEA) due to network topology and anycast limitations, which led to outages. It describes the development of DHCPLB through hackathons and intern projects, highlighting its features like DHCPv6 support, A/B testing capabilities for KEA server updates, MAC address-based overrides, integration with monitoring infrastructure, hot-reloading configuration, BGP advertisement, and two balancing algorithms. The successful rollout of DHCPLB in production demonstrated improved load balancing and simplified testing of DHCP infrastructure, directly contributing to enhanced site reliability.
  52. Scalable and secure access with SSHThis post details Meta's implementation of signed SSH certificates with principals to achieve scalable and secure access to its vast server infrastructure. It addresses the limitations of traditional password and public-key management for large-scale environments, introducing a robust system that leverages certificate authorities for authentication, principals for authorization, and centralized logging for accounting. The post also outlines the implementation details, including the creation of security domains and the configuration of SSH servers to enforce access policies.
  53. Peter O’Hearn elected Fellow of the Royal Academy of EngineeringThis post announces Peter O'Hearn's election as a Fellow of the UK Royal Academy of Engineering for his pioneering work on separation logic. This theory is the foundation for Facebook's open-source static analysis tool, Infer. Infer is highlighted as a system that catches over 1,000 bugs per month before code shipment, saving engineers time and improving user experience. The post details how separation logic enables Infer to model program behavior and reason about independent parts of an application, scaling to large codebases. It also mentions Infer's open-source adoption by companies like Uber and Spotify.
  54. The mobile device lab at the Prineville data centerThis post details the development and scaling of a mobile device lab within the Prineville data center to support automated performance testing of the Facebook app. It describes the iterative hardware design process, from 'sled' and 'gondola' to custom-built EMI-isolated racks capable of holding 32 phones each, and the software challenges overcome using package management and Chef for consistent device configuration and monitoring. The post also outlines future plans for densifying racks, improving iOS testing automation, and extending the platform to other teams.
  55. Making Facebook self-healing: Automating proactive rack maintenanceThis post introduces Aggregate Maintenance Handlers, an enhancement to the FBAR service, to automate rack-level and multi-rack maintenance. It details the decision-making process based on maintenance impact (scope, type, time, duration) and outlines a four-step handler disable/enable process (preflight check, pre-disable, host-level disable, post-disable). It also introduces Dapper for coordinating automated and manual maintenance, and discusses lessons learned from serial and parallel use of disable logic, highlighting the need for smarter, aggregated approaches to avoid capacity issues and race conditions.
  56. Safety Check: Streamlining deployment around the worldThis post details the technical evolution of Facebook's Safety Check feature, focusing on its engineering challenges and solutions. Key contributions include: 1. **Social Graph Activation Algorithm:** A distributed breadth-first search approach leveraging the social graph to quickly identify and notify users in affected areas. 2. **Rate Limiting Service:** Implementation of a new internal rate-limiting service to control job scheduling and prevent alerts caused by excessive job creation during activations. 3. **Initial Propagation Delay Mitigation:** A strategy to bump counters without scheduling jobs for the first few seconds of a launch to ensure counter synchronization across data centers. 4. **Proactive Testing System:** Development of a "shadow launch" system to perform simulated activations every 12 hours for continuous testing and preparedness. 5. **Automated Deployment Tool:** Creation of an internal tool to enable trained teams across time zones to activate Safety Check without engineer intervention. 6. **Crisis Bot for Messenger:** Development of an internal bot for Messenger to automate launch monitoring and provide on-demand data reports.
  57. Python in production engineeringThis post details the extensive use of Python in Meta's production engineering for infrastructure management, platform services, and service configuration. It highlights Python's role in managing hardware lifecycle, network switch setup (TORconfig, FBOSS), core services (Kobold), auto-remediation (FBAR), maintenance scheduling (Dapper), server imaging (Cyborg), and fault detection (machinechecker). It also covers platform services like Job Engine, fbpkg, FBTFTP, and Osmosis, and configuration management using Configerator and Tupperware. The post quantifies Python's usage (21% of infrastructure codebase, millions of lines of code) and notes the increasing adoption of Python 3.
  58. OpenBMC: One board management software for all hardware at FacebookThis post announces the extension of OpenBMC support to Lightning, Meta's new NVMe-based storage platform. It details the challenges of managing NVMe enclosures without a standard like SES and describes the non-traditional BMC design for Lightning, which uses USB, I2C, and PCIe interfaces for communication instead of a sideband network connection. New features like 'fscd' for fan control, 'log-util' for capturing event/error logs, and 'flash-util' for SSD management are introduced. The adoption of OpenBMC across all hardware types (network, compute, and storage) is highlighted as a step towards a more flexible and reliable infrastructure, enabling uniform tools and event/error logs for improved hardware management and fault analysis.
  59. Hardware and firmware attacks: Defending, detecting, and respondingThis post introduces the application of osquery for hardware monitoring to detect firmware and hardware attacks. It details how osquery can be used to gather insights into firmware components, kernel drivers, bootloaders, and other low-level system artifacts. The post highlights the importance of extending visibility to the firmware layer for vulnerability management, incident response, and IT compliance, and provides examples of firmware attack vectors and opportunities for defense.
  60. Improving software RAID with a write-ahead logThis post introduces a write-ahead log (WAL) mechanism to improve the reliability of software RAID 5 and 6 implementations. The WAL addresses the 'write hole' by ensuring data and parity are logged before being written to RAID disks, allowing for consistent recovery after power failures. It also improves rebuild performance by limiting the scope of recovery to the log data size and enhances write performance by aggregating smaller writes into full stripe writes. The implementation uses an SSD as the log disk and has been merged into the Linux 4.4 kernel.
  61. Instrumenting meetings at FacebookThis post details the development and deployment of several internal tools at Facebook to improve the efficiency of in-person meetings and office navigation. These tools include 'Wayfinders,' interactive touchscreen digital maps that dynamically update with real-time information on employee locations and conference room availability, and 'Room Tools,' digital calendars outside conference rooms for on-the-fly booking and issue reporting. A key engineering contribution is the integration of motion sensors in conference rooms with calendar systems to detect and automate the cancellation of unused booked rooms, reclaiming approximately 434 hours of meeting room inventory per week. The analysis of combined data from room calendars, motion sensors, and videoconferencing utilization is used to inform office space design and manage real estate portfolio growth.
  62. Reducing FOOMs in the Facebook iOS appThis post details Meta's efforts to reduce out-of-memory (OOM) crashes in the Facebook iOS app. It describes the process of identifying OOMs through logging and process-of-elimination, profiling memory usage to pinpoint issues with UIWebView, migrating to WKWebView to offload memory usage to a separate process, optimizing allocation rates, and implementing an in-app memory profiler using Objective-C method swizzling to track object allocations and detect leaks. The post highlights the importance of addressing OOMs as a blind spot in traditional crash reporting and the impact of these efforts on app reliability.
  63. OpenBMC for server: Porting and supporting new features for “Yosemite”This post details the porting and enhancement of OpenBMC for the multi-node server platform 'Yosemite'. Key contributions include removing the insecure RMCP+ protocol, adding SSH and REST API support for management and debugging, implementing IPMI FRUID support for various Field Replaceable Units, developing a sensor monitoring daemon, enabling multi-node support by updating software modules and utilities to handle multiple server boards, enhancing the Ethernet driver with NCSI support, adding I2C driver secondary support for IPMB communication, implementing an IPMB framework for BMC and Bridge-IC communication, adding IPMI SDR support with a caching daemon, implementing discrete signal monitoring for error conditions, and developing utilities for FRUID information, Serial-Over-LAN, power control, and sensor readings. A REST API was also introduced for programmatic management.
  64. Osquery: Introducing query packsIntroduced query packs for osquery, enabling users to group related queries by function or problem domain for easier distribution and management. These packs leverage osqueryd's scheduler for automated execution and data collection. Meta is also open-sourcing many of its internal query packs for incident response, vulnerability management, and OS X attacks, and encouraging community contributions.
  65. Open-sourcing Facebook Infer: Identify bugs before you shipThis post details the open-sourcing of Facebook Infer, a static program analyzer used internally to identify bugs like null pointer access and resource/memory leaks in mobile applications before shipping. It introduces the novel techniques of separation logic and bi-abduction for efficient, scalable analysis of large and rapidly evolving codebases, enabling developers to fix hundreds of potential bugs monthly.
  66. How RocksDB is used in osqueryThis post details how osquery utilizes RocksDB as its embedded persistent key-value store for storing and accessing operating system data. It explains how RocksDB enables osquery to perform fast, persistent storage, which is crucial for its event-based pub/sub system. The post outlines the architecture of osquery's event system, including publishers, subscription contexts, and event contexts, demonstrating how RocksDB facilitates the tracking of state changes and the efficient handling of hardware events.
  67. Mobile performance: Tooling infrastructure at FacebookThis post introduces CT-Scan, a performance monitoring and prediction platform designed to detect and prevent regressions in complex mobile applications. It details the design principles of pattern detection and diagnostics, and outlines the system's implementation across development, staging, and production environments. Key features include automated A/B testing for code changes, continuous experiments on the main branch, detailed profiling during triaging, and low-impact real-world performance counter sampling. The post also highlights the system's scalability and extensibility, mentioning its distributed nature, ZooKeeper-based coordination, and the development of a mobile device rack.
  68. Introducing “OpenBMC”: an open software framework for next-generation system managementThis post introduces OpenBMC, an open-source Linux distribution for Baseboard Management Controllers (BMCs). It details the motivation for building a custom BMC stack due to the limitations of closed-source vendor solutions, highlighting the need for reliability and rapid iteration. The post describes the architecture of OpenBMC, including its bootloader, Linux kernel, open-source packages, and board-specific components, all built using the Yocto Project. It outlines the three-layer structure (common, SoC, board) for customization and announces the release of the OpenBMC repository on GitHub, inviting community contributions for system management applications, SoC support, and board-specific improvements.
  69. Stetho: A new debugging platform for AndroidIntroduced Stetho, an open-source debugging platform for Android that integrates with Chrome Developer Tools. Stetho allows developers to inspect network activity and application data, and extends Android's `dumpsys` functionality with custom plugins for more convenient debugging and internal settings manipulation.
  70. Favorite Hacks of 2014This post details several hackathon projects that contributed to infrastructure improvements. The 'Autoscale' hack developed a power-efficient load balancing system that achieved 10-15% average power savings. 'Drone hacks' explored experimental Android technologies and community building, with some projects focusing on voice-controlled drones and algorithmic stabilization. The '@Scale app' was built using Parse to help attendees navigate a technical conference. The 'osquery' project, launched in October 2014, provides real-time infrastructure insight and malware detection capabilities, with a hackathon challenge to build new tables.
  71. Security @Scale 2014 RecapThis post details the "Security @Scale 2014" conference, highlighting Meta's and other industry leaders' efforts to build scalable security solutions. It showcases advancements in bug bounty programs (HackerOne), vulnerability management (Square's Report Card, Doorman, Sting), scalable rule engines for spam fighting (Facebook's Haxl), widespread SSL adoption (Twitter), host intrusion detection (Facebook's osquery), and incident response strategies (Facebook's homebrew IR and Network Security Monitoring, GitHub's ChatOps). This contributes to the understanding of how Meta addresses large-scale security challenges and collaborates with the industry to improve overall online safety.
  72. Introducing osqueryIntroduced osquery, a framework that exposes an operating system as a high-performance relational database, allowing SQL-based queries for low-level operating system monitoring. Key components include osqueryi (interactive query console) and osqueryd (host monitoring daemon). The system is cross-platform (Ubuntu, CentOS, Mac OSX) and designed for modularity and extensibility, with a focus on security and operational insight.
  73. Debugging file corruption on iOSThis post details the debugging process for a file corruption issue on iOS, leading to a 50% reduction in crash rates. The authors used Hipal and Scuba for data aggregation, identified a potential file overwrite issue by switching to SQLite, and then used a honeypot file and conditional breakpoints with lldb to pinpoint the SPDY network stack as the culprit. They further utilized Fishhook to intercept POSIX write calls and confirmed that the SSL layer was writing to a closed and reassigned socket, causing the corruption. The fix involved synchronizing lifetime handling between the SPDY socket and the SSL layer.
  74. The Mature Optimization HandbookThis post introduces the concept of 'mature optimization' as a disciplined approach to performance tuning, emphasizing measurement, cost-benefit analysis, and deferring optimization until systems are mature. It cautions against premature optimization and highlights that significant gains are often concentrated in a small portion of code, discoverable through careful measurement. The post also touches upon performance as a feature when resources are limited or post-ship changes are difficult.
  75. Windex: Automation for database provisioningThis post introduces Windex, an automated system for database provisioning, which complements existing automation for detecting and resolving MySQL host issues (MPS). Windex handles the re-imaging, OS installation, and MySQL configuration of hosts, ensuring a pool of spare MySQL server instances is always available. It includes robust validation, retry logic, and failure pattern detection to prevent damage and identify root causes of provisioning failures, thereby enhancing the overall site reliability and efficiency of database infrastructure management.
  76. Fighting spam with pure functionsIntroduced FXL (Feature eXtraction Language), a domain-specific language optimized for spam fighting at Facebook. FXL is a subset of Standard ML designed for efficient data fetching across the graph. Its core innovation is the use of pure functions, which enables automatic optimizations like batching of data requests (leading to a ~20x speedup) and memoization of common expressions, significantly improving performance and enabling rapid adaptation to new spam threats.
  77. Under the Hood: Automated backupsThis post details the three-stage automated backup system for Facebook's large-scale MySQL installations. Stage 1 involves rack backups (RBUs) using binary logs for real-time updates and mysqldump for logical backups, ensuring data safety and enabling rapid promotion of secondary servers and point-in-time recovery. Stage 2 utilizes Hadoop DFS for replicated, long-term storage of backups and binary logs. Stage 3 involves copying backups to discrete storage in a separate region for long-term archival. The system emphasizes rapid failure detection and automated correction, with a monitoring system that scores backup failure severity exponentially with age. Continuous restore testing and the ORC system for self-service restores are also highlighted.
  78. Monitoring cache with ClaspinThis post introduces Claspin, a novel monitoring tool developed by the cache performance team. Claspin utilizes heatmaps to visualize the status of cache systems (Memcache and TAO) in an easily interpretable format, addressing the growing complexity of Facebook's infrastructure. It encodes 'tribal knowledge' about metric 'hotness' and thresholds to quickly identify potential problems, enabling faster troubleshooting and proactive issue resolution. The post details the evolution of the tool from command-line interfaces to sophisticated, real-time visualizations, significantly enhancing the ability to monitor and manage cache health.
  79. Building and testing at FacebookThis post introduces Facebook's approach to product testing and launching, emphasizing the importance of iterative development and data-driven decision-making. It details the use of a sophisticated tool called 'gatekeeper' to run hundreds of A/B tests daily on random samples of users or entire markets. The post highlights the 'ship early, ship often' philosophy, advocating for launching imperfect products to gather real-world data and iterate quickly. It provides an example of a failed chat bar feature launch and subsequent correction, underscoring the value of empirical data over internal debate. The core contribution is the articulation of a systematic, data-informed process for product development and improvement.
  80. Release engineering and push karma: Chuck RossiThis post introduces the concept of 'push karma' as a mechanism to manage risk associated with frequent code pushes. It details the daily push process, the role of release engineering in supporting rapid iteration, and the importance of test automation and monitoring dashboards. The post also highlights the cultural aspect of engineers taking ownership of their changes from inception to production.
  81. Moving Fast and Staying Reliable: Mary PimenovaIntroduced the Site Performance team's methodology for identifying and tracing site speed and reliability issues using machine learning on page load data. This involves collecting metrics like query cache times and log messages to detect trends and pinpoint the root cause engineer.
  82. Making Facebook Self-HealingIntroduced FBAR (Facebook Auto-Remediation), a system that automates the remediation of server and software outages. FBAR consists of an Alert Fetch Loop to query monitoring systems, a Job Queue, and a Job Engine that executes remediation plugins. These plugins interact with infrastructure APIs to perform actions like removing servers from production, flagging hardware for replacement, and re-enabling repaired machines. FBAR currently manages over 50% of Facebook's infrastructure and has significantly increased service reliability.
  83. Keeping the Site Reliable While Moving FastThis post details the initial efforts in measuring and monitoring user-visible errors across server and client sides. It describes logging errors with Facebook Scribe (1% sampling rate), including details like browser type, user country, and stack traces. The massive data (600GB/day) is processed using Apache Hive to generate trends and compute reliability metrics such as error-free page requests and successful service requests. Specific examples of uncovered issues and their fixes are provided, including client-side JavaScript event alignment, Chat connection and message sending problems, database access errors due to lack of local replicas, and back-end service robustness improvements through throttling and rate limiting. The post also outlines the development of a real-time monitoring system with over 1500 trend lines and the use of machine learning for anomaly detection and alarm generation.
  84. Introducing Login ApprovalsThis post introduces 'Login Approvals,' a new opt-in security feature that implements two-factor authentication using SMS codes sent to a user's mobile phone. It highlights the engineering effort, including intern contributions, to balance security with usability, opting for SMS over app-based or hardware tokens for broader accessibility. The feature aims to protect user accounts from unauthorized access by requiring verification from new or unrecognized devices, with options to save trusted devices and recover access if a phone is lost.
  85. More Details on Today’s OutageThis post details a severe outage caused by an automated configuration verification system that entered a feedback loop with the database cluster. The system incorrectly interpreted a change to a persistent configuration value as invalid, leading to clients attempting to fix it by querying an overwhelmed database cluster. This created a feedback loop where errors from the database caused clients to delete cache keys, further increasing database load. The resolution involved stopping all traffic to the database cluster to allow recovery. The key lesson learned is the need for configuration systems to handle feedback loops and transient spikes more gracefully, prompting exploration of new designs.
  86. Site Reliability Engineering at FacebookThis post introduces the foundational principles of Site Reliability Engineering (SRE) at Facebook, defining its core responsibilities: ensuring the site is up and running 24/7 ('Site'), maintaining feature availability and performance ('Reliability'), and proactively improving systems through automation and performance tuning ('Engineering'). It highlights the team's significant impact with a high user-to-SRE ratio and mentions the use of tools for managing large memcached footprints, provisioning servers, and performance tracking.
  87. Hammering UsernamesThis post details the operational engineering challenges and solutions for launching the username feature to 200 million users simultaneously. It describes the use of Memcached infrastructure for name availability checks and blocked names, a stripped-down registration page to minimize load, strategic launch timing, and the establishment of a 'War Room' for real-time monitoring and response. Key technical strategies included 'dark launching' to pre-test backend functionality under simulated load and the implementation of 'levers' and 'nuclear options' for dynamic throttling of site features (e.g., Facebook Chat, story counts, commenting/liking) to manage unexpected traffic surges.
  88. XHProfLive: Continuous function-level performance stats from productionIntroduced XHProfLive, a system for continuous, function-level performance statistics from production PHP web requests. XHProfLive uses XHProf, a lightweight instrumentation-based callgraph profiler, to gather hierarchical profiles including execution times, CPU time, memory usage, and call counts, broken down by callers. It aggregates these profiles by dimensions like time, page, and data center, enabling detailed performance diagnosis and regression detection through a rich UI and automated alerts. XHProf was also open-sourced.