Internet Latency and Network Performance Analysis

7/13/2026

This post details the modernization of the Meta Ads Service by implementing a custom scheduling policy using sched_ext, an open-source, BPF-based extensible scheduling framework. This was done to address latency regressions caused by a Linux kernel upgrade and to optimize the Ads delivery workload. The custom policy soft-partitions CPUs, prioritizes latency-critical threads, and dynamically adjusts pool sizes based on load, improving L3 cache locality. The implementation resulted in a 28% reduction in ads retrieval stage p99 latency, 3.28 MW power savings, and a 1.1% increase in ads ranked. The post also highlights the benefits of sched_ext as a parallel optimization path, enabling independent deployment of scheduler improvements as user-space BPF program updates, and its upstreaming into Linux for broader ecosystem use.

6/22/2026

This post details Meta's adoption of the AV1 video codec for real-time communication (RTC), focusing on the technical challenges and solutions implemented to achieve scale and improve call quality. Key contributions include the development of a low-complexity AV1 encoder preset comparable to H.264/AVC, the selection of the dav1d decoder for its efficiency, and strategies for reducing binary size. It also describes the use of an ML-based device eligibility framework to identify capable Android devices, enabling wider AV1 coverage.

4/9/2026

Introduced a dual-stack architecture for WebRTC within a monorepo, enabling A/B testing of legacy and upstream versions. Developed a shim layer to provide a unified API and dispatch calls at runtime, minimizing binary size increase. Implemented automated renamespacing scripts to resolve symbol collisions caused by statically linking two versions of the library. Utilized C++ using declarations for backward compatibility and a template-based helper library for runtime dispatch. Developed a code generation system using AST parsing to automate shim creation. Rewired application references to shim equivalents and duplicated internal components using C++ macros and build machinery to handle deep dependencies.

10/16/2025

This post details Meta's '10X Backbone' initiative to scale its Backbone network for AI workloads. It introduces DC Metro Architecture for simplified and standardized DC connectivity, IP Platform Scaling (scaling up with larger chassis and faster interfaces like 800G, and scaling out by adding planes or devices), and IP/Optical Integration using ZR technology to significantly reduce power consumption and simplify deployments. For AI cluster expansion beyond data center campuses, solutions for different reach requirements (FR, LR, and ZR plugs with DWDM) are presented.

9/26/2025

This post details the @Scale: Networking 2025 recap, highlighting the critical role of network engineers in building and operating the infrastructure for AI. It emphasizes the rapid evolution of AI workloads and the corresponding network demands, including the need for co-designing networks with the AI stack, ensuring high reliability, and continuous innovation. The post discusses Meta's investments in AI infrastructure, such as gigawatt-scale clusters and transoceanic fiber cable systems, and the adaptation of networking for diverse AI workloads like large-scale training, mixture-of-experts, and distributed inference. It also outlines the two major themes from the conference: underlying physical network infrastructure and higher-layer, model-oriented talks.

9/24/2025

This post details Meta's collaboration with Vodafone and Google on a white paper recommending the adoption of the AV1 video codec for mobile devices. It highlights AV1's ability to enhance video compression by 30% over H.264 and VP9, and addresses the challenge of limited hardware support in low- and mid-tier smartphones. The post recommends SoC vendors evaluate AV1 hardware adoption and consider software-based decoders as an alternative. It emphasizes the benefits of AV1 for reducing network capacity, optimizing compute and edge cache resources, and saving energy costs, ultimately improving the streaming video experience for a wider range of users.

2/3/2025

This post details Meta's approach to handling leap seconds within its Precision Time Protocol (PTP) infrastructure, contrasting it with traditional NTP methods. It introduces 'self-smearing' via the fbclock library, which shifts time by 1 nanosecond every 62.5 microseconds, and discusses the trade-offs of using UTC over TAI for timekeeping. The post also advocates for the industry-wide freeze of leap seconds to simplify time synchronization and improve precision.

1/10/2025

This post details Meta's ILA Evo effort to reimagine fiber optic in-line amplifier (ILA) sites. It outlines the historical context of ILAs, their current design constraints, and the new requirements for ILA Evo, focusing on speed, cost, and power efficiency. The post describes the engineering efforts in several categories: lightweight building systems (FRP/GRP panels), low-concrete foundation designs (helical steel screw piles), manufactured owner-furnished equipment (MOFE) for ISP installation, ballistic protection, advanced cooling technologies (passive/liquid-based cooling, higher temperature set points), and modern backup power alternatives (H2 fuel cells, molten salt batteries). It also discusses the macro benefits of ILA Evo, including supply chain advantages and the ability to stockpile components.

12/18/2024

This post details how the Threads iOS team addresses performance challenges during rapid growth. It introduces key metrics like %FIRE, TTNC, and cPSR for monitoring image rendering, app startup/feed loading, and content publishing success rates, respectively. The post also describes the development and use of a new tool called SLATE for consistent measurement of navigation latency. A case study highlights the adoption of GraphQL for network requests, validated by SLATE, to improve efficiency over the previous REST-based stack.

8/5/2024

This post details Meta's development and operation of large-scale RoCE networks for distributed AI training. It describes the network topology, including dedicated frontend and backend networks, AI Zones with two-stage Clos topologies, and aggregator training switches for inter-building connectivity. The post also discusses routing evolution, moving from ECMP and path pinning to Enhanced ECMP with queue pair scaling to address low entropy and burstiness in AI workloads. Finally, it outlines the shift in congestion control from DCQCN to receiver-driven traffic admission for 400G deployments, highlighting the use of collective libraries and RoCE transport.

6/13/2024

This post introduces MLow, Meta's new low bitrate audio codec designed to improve real-time communication (RTC) quality, particularly for users on slow connections or low-end devices. MLow achieves two-times-better quality than Opus at 6kbps WB (POLQA MOS 3.9 vs 1.89) while maintaining 10 percent lower computational complexity. It utilizes a split-band encoding approach with advancements in excitation generation, parameter quantization, and coding schemes, building on CELP principles. MLow also enables more effective Forward Error Correction (FEC) strategies, improving audio quality in packet loss scenarios.

5/22/2024

This post details Meta's proactive approach to post-quantum readiness for TLS, addressing the future threat of quantum computers to current public-key cryptosystems. It outlines the company's strategy for migrating to post-quantum cryptography (PQC), including the adoption of hybrid key exchange (Kyber with X25519) and enhancements to the Fizz TLS library using liboqs. The post also highlights significant challenges encountered, such as large packet sizes affecting TLS resumption and TCP Fast Open, and a multithreading race condition in liboqs, along with the solutions implemented or being explored to overcome them.

3/20/2024

This post details the adoption of a machine learning (ML)-based approach for optimizing bandwidth estimation (BWE) and congestion control in Meta's real-time communication (RTC) systems. It describes the transition from a tuned Google Congestion Controller (GCC) to an ML model that characterizes network types (e.g., random vs. congestion-induced packet loss) and predicts future network conditions. The post outlines the model architecture, which combines LSTM for time-series data and dense layers for non-time-series data, and details the process of generating training samples and tuning parameters offline. Experiment results show significant improvements in reliability and quality metrics, such as reduced connection drop rates and peer video freeze percentages.

3/20/2024

This post details Meta's adoption and implementation of the AV1 video codec for improving real-time communication (RTC) video quality across its apps. It highlights the challenges of low-bandwidth networks and the benefits of AV1 over older codecs like H.264, focusing on its royalty-free nature and superior compression efficiency. The post also discusses technical solutions for improving video quality, including custom video scalers and region-of-interest encoding, and elaborates on AV1's specific features like palette mode, intra-block copy, and reference picture resampling (RPR) for enhanced screen content encoding and dynamic resolution adjustments. It also addresses the trade-offs of AV1, such as increased CPU and battery usage, and the strategies employed to mitigate these challenges, including seamless codec negotiation.

2/7/2024

Introduced Simple Precision Time Protocol (SPTP) as a simplified, more reliable, and resource-efficient alternative to PTPv2 for clock synchronization in Meta's data centers. SPTP reduces network communication by simplifying the exchange between client and server to three packets and eliminating the need for state machines on both ends. It also enhances reliability through a clock ensemble mechanism that uses Chauvenet's criterion for outlier detection and a weighted average of valid clocks to determine accurate timing information and a window of uncertainty.

11/15/2023

This post details Meta's network infrastructure evolution to support AI workloads, including GenAI training and inference. It highlights the transition to GPU-based training, the deployment of RoCE-based network fabrics with CLOS topologies, scaling RoCE networks with RoCEV2 transport, and the implementation of centralized traffic engineering for AI training clusters. It also introduces network observability tools like ROCET and PARAM benchmarks within the Chakra ecosystem, and the Arcadia simulator for end-to-end AI system performance simulation.

9/7/2023

Introduces Chakra execution traces, an open graph-based representation of AI/ML workload execution, designed for benchmarking and network performance optimization. Details limitations of traditional AI benchmarking and explains how Meta leverages Chakra for benchmarking (using Mystique and PARAM) and performance optimization through visualization of collective message sizes. Outlines future plans including gathering pre-execution traces, using AI to generate representative traces, and industry collaboration through MLCommons to establish Chakra as a standardized framework.

4/17/2023

Introduced Millisampler, a new fine-grained network traffic characterization tool. Millisampler uses an eBPF-based tc filter to collect network metrics at configurable high-granularity timescales (100us, 1ms, 10ms), including traffic volumes, active flows, ECN marks, and retransmissions. It employs per-CPU variables to avoid locks and periodic sampling to minimize overhead. The tool has been used to debug network performance issues and characterize traffic patterns, contributing to improved network efficiency and reliability.

4/11/2023

This post details Meta's adoption and deployment of the xHE-AAC audio codec for delivering high-quality audio at scale on Facebook and Instagram. It explains the technical benefits of xHE-AAC, including integrated loudness management and adaptive bit rate audio (ABR) with immediate playout frames (IPFs) for seamless transitions. The post also outlines the deployment process, involving Fraunhofer's encoder SDK and shaka-packager, and highlights how loudness metadata and client-side configuration enable adaptive audio experiences.

2/21/2023

This post details Meta's implementation and deployment of the AV1 video codec for Facebook and Instagram Reels. It covers the technical rationale for choosing AV1, the optimization of SVT-AV1 encoders and dav1d decoders for global-scale deployment, and the integration of advanced adaptive bit rate (ABR) streaming techniques using a two-stage hybrid hardware/software encoding approach. The post highlights the trade-offs between encoder quality and complexity, the challenges of client-side decoding on mobile devices, and the specific implementation details for Reels, including the use of convex hull optimization for rate-distortion curves.

2/7/2023

This post details significant upgrades to Meta's global basemap infrastructure over the past two years. Key technical advancements include the launch of dark-mode variants and raster-first basemaps, which improve rendering speed and adaptability across devices. The basemap now utilizes the Daylight distribution from OpenStreetMap (OSM) and a new Earth Tables schema for simplified OSM tag processing. AI-based translations are being implemented for map label readability, and high-resolution pavement data and tree locations are being integrated in select cities. Cartographic design has been refined with a softened color palette and deprioritization of roads to emphasize community spaces and popular places. Building features have been re-engineered, with color-coding for "areas of interest" and transportation buildings. New iconography has been introduced for parks, universities, and entertainment features, and existing icons for airports and train stations have been updated. Future work includes exploring global land cover, topography, and building 3D geospatial map data for the metaverse and digital twins.

1/27/2023

This post details Meta's advancements in optimizing large-scale networks, focusing on data center networking (FBOSS, SAI), WAN optimizations (global optical network, submarine cables like Bifrost, Echo, 2Africa), and operational efficiency through fine-grained traffic analysis (Millisampler) and Network SLOs. It also introduces Shiv, a tool for improving Layer 4 routing consistency using consistent hashing to prevent connection breakage.

11/21/2022

This post details Meta's deployment of the Precision Time Protocol (PTP) to achieve nanosecond-level timing accuracy and precision across its networks and data centers. It explains the limitations of NTP for large-scale systems and highlights PTP's advantages in accuracy and precision. The post also describes the engineering challenges in migrating from NTP to PTP, including the development of custom hardware like the Time Appliance and Time Card, and the open-sourcing of these solutions through the Open Compute Project. The benefits of PTP for current products (messaging, gaming, video conferencing) and future technologies (metaverse, AI synchronization) are discussed.

11/21/2022

This post details the implementation of Precision Time Protocol (PTP) at Meta, migrating from NTP to achieve nanosecond precision for system synchronization. It outlines the technical challenges and solutions involved in deploying PTP at scale, including the PTP architecture (rack, network, client), hardware innovations like GNSS-over-fiber antennas and a new Time Appliance with FPGA-based Time Cards and Network Cards, and the development of a custom ptp4u solution to support millions of clients. The post also discusses the use cases for PTP, such as ensuring linearizability in distributed systems, and the monitoring strategies employed.

9/30/2022

This post details the development and rollout of a custom Chromium-based WebView for the Facebook Android app. This initiative addresses issues with outdated system WebViews leading to security risks and app crashes. The custom WebView allows for bundled updates with the Facebook app, ensuring users receive the latest security patches. It also resolves stability issues by avoiding crashes during system WebView updates and improves rendering performance by enabling the GPU process. Meta also highlights its commitment to open-source contributions to Chromium.

9/7/2022

Introduced Network Entitlement, a contract-based network sharing solution that reserves bandwidth per service for a predefined period for each region. This framework addresses challenges in capacity utilization, service isolation, accountability, SLO guarantees, and contextual tax on services by providing isolation for reliability, guarantee and accountability, abstraction of network complexity, observability, and a work-conserving approach. The solution comprises four components: contract abstraction, dynamic SLO-based guarantee service, runtime enforcement system, and policy verification using production drills.

7/25/2022

This post details Meta's support for an industry effort to eliminate future leap second introductions. It explains the technical challenges and risks associated with leap seconds, including system crashes, data corruption, and outages caused by unexpected time jumps or smearing implementations. The post describes Meta's current practice of 'smearing' leap seconds over 17 hours using quadratic algorithms and highlights the complexities of time synchronization across different GNSS constellations and NTP servers. It advocates for replacing leap seconds with technologies that provide a more stable and predictable time reference for digital applications.

6/9/2022

This post details Meta's cloud gaming infrastructure, focusing on edge computing, GPU utilization, and video/audio streaming for low-latency gameplay. It introduces the use of NVIDIA GPUs and Twine for orchestration, container technologies for Windows and Android games, and WebRTC with SRTP for streaming. Key technical improvements include GPU encoding to reduce latency and PCI bus bandwidth usage, and hardware decoding on the player side. Security measures like threat modeling and isolation from core data infrastructure are also discussed. The post connects cloud gaming infrastructure challenges to future metaverse requirements.

10/19/2021

Introduced RTMP Go Away, a mechanism to signal live server connection termination to clients at logical media boundaries, enabling lossless reconnections. This involves spec updates to the RTMP connect packet and a new protocol message type for the go away signal. A reference implementation in FFmpeg allows clients to reconnect at the next keyframe. The rollout is safe, as the go away message is only sent to clients that have indicated support for the new feature.

8/9/2021

Introduced a risk-driven backbone management strategy that uses a simulation system to identify potential failures and quantify network risk with metrics around demand loss, availability, and latency stretch. These metrics, derived from failure modeling based on historical data and Bayesian regression, guide operational decisions for capacity deployment and traffic priority management, enabling the backbone to resiliently withstand stress tests and achieve high service availability and low latency.

6/15/2021

Introduced the 'network hose' model for production backbone network capacity planning. This model shifts from forecasting pairwise traffic matrices between data centers to forecasting aggregate ingress and egress traffic per data center. This simplification reduces planning complexity by an order of magnitude and allows services to reason about network resources more like other consumable entities. The approach leverages identifying critical network cuts (especially simple, geographically relevant ones) to derive a minimal set of reference demand matrices that, when planned for, guarantee coverage for all possible traffic matrices satisfying the hose constraint. This method simplifies operational workflows and reduces false alarms related to traffic surges.

5/20/2021

Introduced a new automated method for managing public peering, replacing a time-intensive manual process. This system allows for faster self-service peering configuration through a dedicated web page, leveraging PeeringDB OAuth for authentication. The automated workflow handles request auditing, configuration of BGP sessions on Facebook's routers, and daily monitoring for session establishment, significantly reducing manual effort and improving efficiency. The post also advocates for PeeringDB OAuth as an industry standard for public peering automation and explores extending this automation to private network interconnects (PNIs).

5/13/2021

This post details the scalable design, software implementation, and operations of Facebook's data center routing design based on Border Gateway Protocol (BGP). It introduces a uniform AS numbering scheme, hierarchical route summarization, and policy configurations for reliable communication. It also describes an in-house BGP agent with a multi-phase deployment pipeline for testing and incremental deployment, including unit testing, emulation, and canary testing.

3/22/2021

This post details optimizations made to the server media ingestion 'configure' endpoint for Instagram Threads to reduce send latency. The primary improvement involved identifying that the endpoint spent significant time waiting on I/O due to a write-through cache database. To address this, a routing rule was implemented to force requests to a faster region when a special HTTP header was present, effectively batching cross-region round trips to the backing store and reducing overall endpoint execution time, leading to a 29% reduction in send latency.

12/30/2020

This post details the adoption and impact of QUIC and HTTP/3 on Meta's internet traffic, showing significant improvements in request errors, tail latency, and other metrics. It also highlights the complete rewrite of Messenger for iOS (Project LightSpeed) resulting in a smaller, faster app with fewer lines of code, and the redesign of the Facebook web app using React and Relay. Additionally, it covers the use of automation and ML for infrastructure hardware maintenance, new tools for engineers (Health Compass, Incident Tracker), and hardware optimization for microservices (SoftSKU, Accelerometer). Connectivity innovations include aerial fiber deployment robots, the 2Africa subsea cable, SuperCell for rural connectivity, and the Evenstar program for affordable broadband.

10/22/2020

This post details Meta's significant advancements in scaling live streaming infrastructure to support millions of concurrent viewers for both user-generated content and professional broadcasts. It highlights the development of new ingestion capabilities with built-in redundancy and failover mechanisms, the scaling of delivery infrastructure to combat the 'thundering herd problem' through request coalescing and cache sharding, and proactive collaboration with ISPs to manage network congestion. The post also outlines future challenges and explorations, such as using LTE-Broadcast for last-mile delivery and the need to optimize for a high volume of simultaneous smaller-reach streams during events like New Year's Eve.

10/21/2020

This post details Meta's implementation and deployment of QUIC and HTTP/3, detailing the development of their own QUIC implementation called mvfst. It highlights the technical challenges encountered, such as application logic regressions affecting static content delivery, CPU efficiency concerns with mvfst for static content, and the tuning of congestion control (BBR) and flow control parameters. The post quantifies the improvements seen in metrics like request errors, tail latency, and video rebuffering (MTBR) across Facebook and Instagram apps, emphasizing the benefits for users in emerging markets. It also discusses the integration of QUIC with existing infrastructure like Proxygen and network load balancers, and the strategy for rolling out the protocol to billions of users.

3/18/2020

This post details Meta's migration from ntpd to chrony for its internal NTP service, achieving accuracy improvements from ~10ms to microseconds. It describes the four-strata architecture of their NTP service, including the use of atomic clocks and GNSS. The post also covers the challenges of time synchronization at scale, the limitations of daemon estimates, and the validation of accuracy using 1PPS signals and specialized hardware. Finally, it announces the open-sourcing of their NTP libraries.

11/25/2019

This post details advancements in network reliability, automation, Anycast content delivery, mesh networks for underserved markets, security performance management, enforcing encryption at scale using BPF and Transparent TLS, improving QUIC CPU performance, offloading connection encryption using SmartNICs, performance tools for Envoy scalability, adaptive cache networks with optimality guarantees, and building edge compute platforms for services like Stadia, as presented at the Networking @Scale Boston 2019 conference.

11/21/2019

This post details a groundbreaking event for a new long-haul fiber route connecting data centers in Ohio, Virginia, and North Carolina. It emphasizes the strategic importance of this infrastructure for maintaining reliable, resilient, and scalable access to Meta's services for its global user base, highlighting a pragmatic approach to network infrastructure investment.

11/18/2019

This post details the evaluation and implementation of the COPA congestion control algorithm within Meta's QUIC transport stack for optimizing video performance on the Facebook Live Android platform. It describes COPA's tunable delay-based approach, contrasting it with CUBIC and BBR v1. The post presents A/B test results demonstrating COPA's superior performance in reducing application-observed RTT and improving goodput, especially in adverse network conditions, compared to CUBIC and BBR v1. It also analyzes transport RTT and retransmission overhead, identifying potential network policing as a factor affecting COPA's performance in tail cases. The contribution lies in the practical deployment and large-scale evaluation of a novel congestion control algorithm to enhance real-time video streaming quality and reduce latency.

11/1/2019

This post introduces and explains the concept of 'delegated credentials,' a new extension to the TLS protocol being developed with industry partners. It addresses the challenges of managing TLS certificates at scale, particularly the security risks associated with long certificate lifetimes and the limitations of traditional revocation mechanisms. Delegated credentials offer a way to significantly shorten certificate lifetimes (to hours or days) by allowing servers to sign their own short-lived credentials using their primary certificate, thereby improving security and reducing reliance on revocation without impacting reliability. The post also highlights Meta's implementation of this in Fizz and provides a test endpoint.

9/25/2019

This post details advancements in network reliability and outage management discussed at Networking @Scale 2019. Key contributions include Facebook's BGP++ for scalable routing, Microsoft's Open Network Emulator (ONE) for pre-deployment testing of network changes, Cloudflare's lessons learned from bootstrapping 1.1.1.1, and Facebook's operational experience with its SD-WAN network (Express Backbone). It also covers Facebook's infrastructure for enforcing encryption at scale using BPF programs and Transparent TLS (TTLS), and Netflix's Probnik client-side network measurement system.

6/20/2019

This post details Meta's contributions to the web platform and browser technologies to improve web application performance. Key contributions include the proposal and implementation of the `isInputPending()` API to improve JavaScript scheduling and responsiveness, the integration of retired instruction counts into Chrome's performance tracing system for more reproducible benchmarking, the proposal and implementation of a native JavaScript Self-Profiling API for low-overhead JavaScript execution analysis, and the development of BinAST, a binary encoding format for JavaScript ASTs to reduce parsing overhead. These efforts aim to address the performance disparity between web and native applications by improving JavaScript code size scaling and web introspection capabilities.

5/29/2019

This post details Meta's migration of its internal data center service encryption from Kerberos to TLS. It highlights the challenges of managing Kerberos at scale, leading to the adoption of TLS with X.509 certificates for improved operability and performance. Key contributions include the design principles for managing the security-operability trade-off, the use of session tickets for efficient symmetric key cryptography, and the development of an internal certificate authority (ICA) for robust identity provisioning during container setup, ensuring end-to-end defense in depth for microservices.

5/28/2019

This post details the evolution of Meta's DHCP infrastructure, moving from ISC-DHCP to Kea, and finally to a new DHCPLB server architecture. The key contribution is the development of a multithreaded DHCPLB server that replaces the single-threaded Kea, addressing performance bottlenecks related to the Recv-Q buffer and backend calls. This new architecture significantly improves throughput, reduces server count by 10x, and enhances iteration speed. The post also highlights the development of a new Go-based DHCP library and the open-sourcing of the DHCPLB framework, contributing to the broader community.

4/22/2019

This post details the development and contribution of the isInputPending API to the Chrome browser. The API addresses the trade-off between fast page loading and quick event response by allowing JavaScript to check for pending input events before continuing long-running tasks. This prevents user interactions from being queued for extended periods, improving overall web responsiveness. The post outlines the problem of JavaScript queueing time, the proposed solution, the technical implementation details of the API (hooking into the compositor-side input queue), and the process of contributing it to a web browser through an origin trial.

12/21/2018

This post details Meta's pilot project with Cloudflare to implement DNS over TLS (DoT), encrypting DNS traffic between Cloudflare's 1.1.1.1 resolvers and Meta's authoritative name servers. It analyzes the feasibility, latency overhead, and computing overhead of DoT in a production workload. The pilot demonstrated that initial connection latency is amortized through TLS session resumption, resulting in p99 DNS latency comparable to UDP. The post also discusses the historical context of DNS security, the evolution of standards like DNSSEC and DoT, and the importance of operator input in protocol development. Future optimizations with TLS 1.3 and TCP Fast Open are mentioned.

10/19/2018

Introduced MobileLab, a new system for highly accurate mobile performance testing designed to prevent regressions as small as 1 percent. MobileLab achieves this by establishing a deterministic testing environment through several key technical innovations: locking CPU and GPU frequencies to fixed settings, using an HTTP proxy to simulate consistent network responses, managing application on-disk state across trials, remounting data directories to RAM disks for consistent disk I/O, resetting device clocks, dismissing crash dialogs, and replacing logcat tailing with direct socket communication for metric reporting. This system has improved confidence intervals by 7x and reduced false positives by 75 percent compared to previous methods.

9/4/2018

This post introduces FBOSS, an open-source switch software developed to manage large-scale, rapidly growing networks. It highlights how FBOSS, built with a software service approach and modular design, has enabled a 30x increase in deployments and facilitated the implementation of IPv6 and Fabric Aggregators, directly impacting network agility and performance.

8/6/2018

This post details the development and deployment of Fizz, a C++ TLS library, to implement TLS 1.3 at scale. It highlights Fizz's features such as asynchronous I/O, scatter/gather I/O for zero-copy operations, and support for 0-RTT early data with replay protection. The post also discusses the security benefits of Fizz's explicit state machine design and the challenges faced during deployment, including middlebox interference, and the solutions implemented. Fizz has secured over 50% of Meta's internet traffic and is open-sourced.

6/21/2018

This post recaps the 2018 Networking @Scale conference, highlighting technical discussions and open-source contributions. Key technical contributions include the open-sourcing of Katran, a scalable L4 load balancer leveraging XDP and eBPF for improved performance and CPU efficiency, and the sharing of IPv6 adoption data. Technical discussions covered zero-touch provisioning with Vending Machine, network resilience modeling using ML at Microsoft, challenges in at-scale optical networks, HyperPlane as a foundational system for AWS networking services, and Edge Fabric for managing egress load. Blizzard shared application perspectives on network realities for online gaming.

6/6/2018

This post details the growing adoption of IPv6 across the U.S. and other countries, highlighting its performance benefits and the critical need for migration due to IPv4 address exhaustion. It shares specific data on Facebook's IPv6 traffic, including milestones in the U.S. and growth in countries like India, Taiwan, Mexico, and Vietnam. The post also discusses technical optimizations like the 'Happy Eyeballs' algorithm and its impact on user experience, encouraging further industry-wide adoption.

5/22/2018

This post introduces Katran, a new software-based network load balancer for Facebook's infrastructure. Katran leverages eXpress Data Path (XDP) and the eBPF virtual machine to reengineer the forwarding plane, enabling it to run on commodity Linux servers and coexist with other services. This design improves performance and scalability by handling packets earlier in the network stack, reducing inefficiencies like busy loops, and allowing for colocation with backend servers. The first-generation L4LB relied on IPVS and separate machines for load balancing and backend services, which had drawbacks in terms of vulnerability to load increases and failure domain separation. Katran addresses these by offering a more flexible and efficient software-based solution.

5/21/2018

This post introduces Zero Touch Provisioning (ZTP) and the Vending Machine system, a comprehensive workflow framework developed to automate the provisioning and deployment of Facebook's global backbone networks. It details the challenges of rapid growth in heterogeneous network environments and how Vending Machine, with its Python agents and modular 'steps', addresses these by replacing manual Method of Procedure (MOP) documentation with automated, iterative workflows, significantly increasing deployment speed and reliability.

4/10/2018

This post details Meta's practical testing of the AV1 video compression codec against x264 and libvpx-vp9. It outlines the methodology, encoder configurations, and experimental results, demonstrating AV1's superior compression efficiency (30-50% better than alternatives) at the cost of significantly longer encoding times. The findings are presented with detailed metrics (BD-rate, PSNR, SSIM, encoding time ratios) and are intended to inform engineers about AV1's potential for real-world production environments.

3/20/2018

Introduced the Fabric Aggregator, a distributed network system designed to scale network capacity between data center fabrics. This system leverages simple, open building blocks like Wedge 100 switches and FBOSS, employing a disaggregated approach with a 2-layer cross-connect architecture (downstream for east/west traffic, upstream for north/south traffic). It allows for independent scaling of traffic capacities and ensures reliability through isolated failures due to independent subswitch operation. The design supports both single-rack and multi-rack deployment types with various cabling configurations.

3/19/2018

This post recaps the Performance @Scale 2018 conference, highlighting technical discussions and presentations on managing performance challenges at scale. Key technical contributions and discussions include: Andre Vachon (Microsoft) on Execution Graphs for distributed trace processing and performance regression detection in Azure VM deployments; Delyan Kratunov (Facebook) on understanding mobile app performance in the wild and the open-source release of Profilo; Yang Yang and Ritesh Maheshwari (LinkedIn) on ThirdEye, a generic anomaly detection platform for time series metrics, applied to real user monitoring data; Rico Mariani (Facebook) on iOS VM and loader considerations impacting application startup time; Brien Colwell (Headspin) on a new test infrastructure for automated mobile performance tests in real user conditions; Guy Cirino (Netflix) on building a Magic Modem for global network emulation; Xiaowei Jiang (Alibaba) on scaling real-time infrastructure for global shopping holidays; and Nick Gavalas (Facebook) on efficient static resource delivery mechanisms for Facebook on the web.

3/13/2018

Introduced Profilo, a high-throughput, mobile-first performance tracing library, now open-sourced for Android. Profilo enhances the ability to collect and analyze detailed telemetry from mobile apps in production, improving the diagnosis of performance regressions by providing precise root-cause analysis for issues as small as tens of milliseconds of CPU time. It enables new causality analyses and more precise understanding of metrics like 'scroll fluidity' and 'app responsiveness'. Key technical contributions include a custom Java stack unwinder that bypasses official Java APIs for efficiency and accuracy, and the capture of 'atrace' (systrace) telemetry in production to analyze VM and Android framework behavior, garbage collection, I/O operations on the UI thread, and lock contention.

2/12/2018

This post details the Production Engineering team's approach to supporting global events on Facebook, specifically focusing on the planning and infrastructure required for Facebook Live during New Year's Eve. It outlines the three categories of load variance (routine, spontaneous, planned), the architecture of Facebook Live, key resource metrics (network, CPU, storage), and the load metrics used for planning (total broadcasts, peak concurrent broadcasts, dependent system load). The post describes the process of estimating traffic using historical data and predictive modeling, scaling strategies like hardware allocation and segment consolidation, and load testing methods (artificial load, synthetic tests, shadow traffic). A specific example of an unexpected issue with health checks impacting routing hosts is also highlighted.

11/15/2017

This post introduces Open/R, an open-source, extensible network routing platform designed to accelerate innovation in network functions and applications. It highlights Open/R's use in Facebook's backbone and data center networks, its support for diverse network topologies and hardware/software systems, and its features like automatic IP prefix allocation and fast convergence. The post also emphasizes the platform's applicability across different network domains (wireless backhaul, global backbone, data center fabrics) and its modern, software-centric design approach, moving beyond traditional hardware-constrained protocols. The open-sourcing of Open/R and its associated emulation framework aims to foster collaboration and adoption within the broader networking community.

8/28/2017

This post introduces FBNet Command Runner (FCR), an open-source command runner for network devices. FCR addresses the challenges of automating operations on a large, multi-vendor network by providing a unified Thrift interface for running commands, abstracting away vendor-specific details and connection management. It is integrated with the Robotron system for active monitoring and can also be used for provisioning and deployment, significantly improving the efficiency and reliability of network operations at scale.

8/21/2017

This post details the development and implementation of Edge Fabric, a system designed to improve egress traffic steering at the network edge. It addresses limitations of traditional BGP by introducing capacity and performance awareness. Key components include a BMP collector and traffic collector for data collection, an allocator that projects interface utilization and generates overrides, and a BGP Injector service to enact these overrides. The system aims to optimize network traffic performance by overcoming BGP's static decision-making and lack of congestion awareness.

5/18/2017

This post recaps the third annual Networking @Scale conference, highlighting technical talks from various companies including Facebook, Alibaba, Amazon, Dropbox, Equinix, Google, Microsoft, and Netflix. Specific technical contributions from Facebook include details on Express Backbone (a private SD-WAN long-haul network for data center interconnection), Robotron (a scalable system for automating and scaling the edge network), Facebook CDN innovations (cache priming for live video, ML-based predictive cache priming), Zero Protocol and TLS 1.3 deployment efforts and challenges, and internet-scale virtual networking using IPv6 ILA. Other companies shared insights on exabyte-scale storage networks, high-performance packet processing, global content delivery logistics, certificate transparency, IPv6 delivery on AWS VPC, IPv6-only corporate networks, high-altitude platforms for rural backhaul, and custom hardware/network design.

5/1/2017

Introduced the Express Backbone (EBB), a new long-haul network designed to separate cross-data center traffic from internet-facing traffic. EBB employs a hybrid traffic engineering model combining distributed control agents (Open/R) with a central controller. Key components include a BGP-based route injector, sFlow collector for traffic matrix estimation, a traffic engineering controller for path computation, Open/R agents for IGP and messaging, and LSP agents for programming forwarding tables. The design draws inspiration from data center network fabrics, using four parallel topologies ('planes') for incremental deployment and A/B testing. The controller's path allocation algorithm can optimize for latency, congestion, or scheduling priorities. Segment routing is used for simplified programming and improved robustness.

3/22/2017

This post details advancements in video streaming technologies discussed at the @Scale 2017 conference. Specific contributions include improvements to Facebook's Dynamic Streaming system for 360 video, development of spatial audio rendering methods for large-scale consumption, and end-to-end considerations for low-latency live stream delivery for Facebook Live. It also highlights research in AI for video understanding and efficient encoding for mobile video.

3/8/2017

This post details Meta's development and deployment of 100G single-mode optical transceivers for data center interconnectivity. It explains the motivation for moving to single-mode fiber to future-proof the fiber plant and reduce recabling costs. The post outlines the challenges of single-mode transceivers (precision alignment, higher cost, power consumption) and how Meta optimized the CWDM4 MSA specification (CWDM4-OCP) by relaxing parameters like reach, link budget, and operating temperature to meet data center needs. It also discusses strategies for mitigating technology and deployment risks, including considering multiple technology platforms, rigorous testing, and working with suppliers to achieve rapid production ramp-up for large-scale deployments.

1/27/2017

This post details the development and deployment of Meta's experimental Zero protocol, a 0-RTT security protocol built over TCP based on QUIC's crypto protocol. It addresses the high latency of TLS 1.2 handshakes, particularly in emerging markets, by aiming for zero additional round-trip latency for security. Key technical contributions include modifications to QUIC's key derivation to prevent vulnerabilities, in-band server config rotation for efficient updates, retry safety mechanisms, time-bound 0-RTT data with clock skew correction, and adaptations for TCP by removing explicit sequence numbers and adding length fields. The post also discusses API design choices for integrating 0-RTT into Mobile Proxygen, opting for a socket API that maintains existing `connect()` and `write()` calls while introducing an `enableZeroRTT()` function.

1/24/2017

This post details the growing ecosystem around Meta's open networking hardware initiatives, highlighting the contributions of various companies (Cumulus Networks, Big Switch Networks, Barefoot Networks, SnapRoute, Canonical, Cavium, Apstra) in developing software and hardware solutions that leverage Meta's open designs like Wedge and Backpack. It showcases the increasing commercial availability and adoption of these open technologies, emphasizing the benefits of openness and collaboration in advancing the networking industry.

10/31/2016

This post details presentations from the Networking @Scale Boston conference, highlighting advancements in network infrastructure and performance. Key contributions include the introduction of Open/R as a modular routing platform for Terragraph and other Facebook networks, discussions on scaling Facebook Live with techniques for handling throughput bursts and optimizing latency, and insights into scaling traffic configurations. It also covers securing microservices with encryption solutions for Thrift, and the deployment of QUIC for internet-scale experimentation and video streaming.

8/31/2016

Introduced BrowserLab, an automated system for detecting client-side performance regressions in web applications. BrowserLab addresses noise sources like environment variance, dynamic data, and code non-determinism through techniques such as an HTTP proxy for isolating server noise, RAM disks and consistent browser states for reducing client environment noise, a backend proxy for consistent data views, and deterministic random number generation for code. It enables accurate measurement of regressions as small as 20 ms and provides root cause analysis via aggregated tracing data. The system is integrated into the continuous integration pipeline for automated regression detection and bisecting.

6/17/2016

This post details the technical development of Canvas, a system for creating full-screen immersive mobile experiences. It describes the UI challenge of condensing creative requests into a small set of intuitive yet flexible components, the use of high-level primitives like autoplay videos, photos, and carousels, and the underlying infrastructure and rendering engines leveraged from other teams. It also explains the design of a custom API and language for configuring components and defining layouts using GraphQL, the implementation of 'actions' for user interactions and chained actions, and the real-time creative preview mechanism using deep links. Technical challenges related to mobile-first design, screen size abstraction, and loading speed optimization are also discussed.

4/20/2016

Introduced Wangle, an asynchronous C++ networking and RPC library inspired by Netty and Finagle. Wangle provides core asynchronous building blocks (folly/io/async, Futures) and a concurrency framework with CPU and IO thread pools. It utilizes a pipeline architecture for processing socket data through a series of handlers and codecs. ServerBootstrap and ClientBootstrap facilitate the creation and configuration of pipeline-based servers and clients. An example demonstrates building a file streaming server that offloads blocking disk reads to the global CPU pool.

3/9/2016

This post details the opening of designs for the 6-pack and Wedge 100 modular switches, representing a significant step towards supporting 100G connectivity in Meta's data centers. It describes the hardware specifications of Wedge 100 (a 32x100G TOR switch) and 6-pack (a 128x40G modular switch), including their use of Broadcom's Tomahawk ASIC, support for OpenRack v2, COM-E CPU modules, and various QSFP28 and QSFP+ optic transceivers and DAC cables. The post also elaborates on the extensive hardware and network-level testing performed for these switches within the data center fabric, highlighting the increased scalability and performance requirements for fabric switches compared to top-of-rack switches (e.g., 16K routes vs. 1K, 5.12Tbps traffic rate vs. 640Gbps). It details critical test scenarios for functionality (BGP peerings, route policy processing, ECMP) and failure conditions (BGP flapping, link failures, card insertions/removals), demonstrating sub-2-millisecond failure handling and rerouting with minimal packet loss. Furthermore, the post discusses the software architecture of FBOSS running on 6-pack, emphasizing a disaggregated approach where the chassis switch is viewed as a set of connected Wedges rather than a traditional supervisor module model, and highlights the ongoing effort to automate the entire test suite.

2/29/2016

This post details the Performance @Scale 2016 event, highlighting advancements in performance engineering across various companies. Key technical contributions discussed include the use of BPF for low-level Linux performance analysis (Netflix), end-to-end web performance solutions at Facebook (Early Flush, Big Pipe, Bootloader), automatic regression triaging at Facebook (AutoTriage, Stack Trace Finder, Pushed Commit Search), client-side performance analysis with PerfView (Microsoft), high-performance networking in Chromium (Google histograms, QUIC protocol), mobile performance telemetry with Loom (Facebook), and visualization techniques for Real User Monitoring (RUMv2) at LinkedIn. The event emphasized the importance of shared learning and best practices in making technology fast and efficient at scale.

2/18/2016

Introduced NetNORAD, a system for end-to-end network probing using UDP probes to measure packet loss and latency, independent of device polling. Developed pinger and responder processes for servers, and the fbtracert utility for path analysis. Implemented data processing pipeline using Scribe and Scuba for real-time logging and analysis, with an alarming process to detect and alert on network failures within 20-30 seconds. Introduced fault isolation principles based on proximity tagging (DC, Region, Global) and downstream suppression, complemented by fbtracert for more granular path analysis. Open-sourced key components (UdpPinger and fbtracert).

12/3/2015

This post details the engineering challenges and solutions for broadcasting live video to millions of users. It introduces two key technical advancements: 1. Edge caching and request coalescing to handle "thundering herd" problems during live broadcasts, ensuring scalability and reliability by caching HLS segments and queuing requests for cache misses. 2. The implementation of RTMP for reduced latency, moving from an HLS segment-based pull model to a push model with small data chunks for near real-time streaming.

11/20/2015

This post details optimizations made to the Facebook for iOS app's start time, focusing on cold starts. Key contributions include: defining stable metrics for app start (from icon tap to feed loaded), breaking down cold start into request time, network time, and response processing time, identifying initialization as a bottleneck, refactoring feed request dependencies to allow concurrent initialization, implementing UDP priming to pre-fetch data and reduce TCP connection overhead, optimizing story fetching from a batch to a streaming approach (1+1+X), and deferring non-critical background services until after the feed is displayed to reduce contention.

11/3/2015

This post details the progress and future plans of the OCP Networking Project, highlighting the acceptance of new switches from Mellanox and the evolution of open hardware and software in networking. It mentions the introduction of FBOSS and OpenBMC, and the Wedge specification for data center fabrics, emphasizing the community's role in reinventing the networking industry for improved speed, scale, capabilities, and openness.

10/27/2015

Introduced "2G Tuesdays" for Facebook employees to simulate 2G network conditions, fostering empathy and informing product design for users on slow connections. This initiative aims to address the challenges of providing a seamless experience on 2G networks, complementing existing tools like Connection Class and Augmented Traffic Control.

9/14/2015

This post highlights Meta's proactive migration to IPv6, emphasizing its benefits for network performance, management, and end-to-end connectivity. It details observed speed improvements of 10-15% for accessing Facebook over IPv6 and advocates for broader adoption of IPv6 to overcome IPv4 limitations and unlock new technological possibilities.

7/27/2015

This post details the implementation and upstream contribution of an on-demand routing cache for Linux IPv6. This solution addresses scalability issues caused by large routing trees in IPv6 deployments by creating routing caches only when needed, rather than pre-emptively. Key technical contributions include a patch series to the upstream kernel, per-CPU entry optimization for gateway routes to mitigate cache bouncing, and ensuring compatibility with subsystems that expect '/128' routes. The post also highlights performance improvements observed in benchmarks and discusses future work to further optimize IPv6 performance and integrate IPv4/IPv6 routing.

4/13/2015

This post details a study conducted by Meta to measure the efficiency of browser caching for facebook.com, especially in light of their twice-daily release schedule. The study recreated a Yahoo performance research experiment by serving a special image with cache-controlling HTTP headers and logging requests. Results showed that 25.5% of requests were missing the cache, with significant variations across browsers (e.g., Firefox v32+ had skewed results due to a new cache backend). Analysis of cache duration revealed that for desktop users, 50% of requests had a cache that was at most 47 hours old. The findings suggest that while browser cache hit rates have improved since 2007, caches do not stay populated for long, a factor influenced by increasing webpage sizes and frequent code deployments. The post concludes that browser caching remains important and provides best practices for its utilization.

3/26/2015

Introduced and open-sourced two Android frameworks: Year Class and Connection Class. Year Class classifies Android devices based on their hardware specifications (RAM, CPU cores, clock speed) relative to their release year, enabling segmentation for analysis and feature customization. Connection Class segments network quality into four buckets (Poor, Moderate, Good, Excellent) based on real-time bandwidth measurements, using a low-pass filter and geometric mean for accuracy. These tools are used to adjust feature performance at runtime and analyze performance bottlenecks.

3/23/2015

Introduced Augmented Traffic Control (ATC), a tool developed to simulate diverse network conditions (initially attempted with 2G, then pivoted to Wi-Fi based simulations) to allow engineers to test applications under realistic, often suboptimal, network environments. ATC enables simulation of 2G, Edge, 3G, and LTE networks, specifically targeting common connectivities in countries like Brazil, India, Indonesia, Kenya, Nigeria, and the Philippines. The tool was integrated into the campus Wi-Fi network and made available via APIs for automated testing, reducing the need for extensive A/B testing and improving development efficiency for features like Messenger and download functionality. ATC was open-sourced on GitHub.

3/10/2015

This post introduces Facebook Open Switching System (FBOSS) and Wedge, a top-of-rack switch. It details the disaggregation of network hardware and software, with FBOSS acting as a set of applications on Linux rather than a proprietary OS. The post highlights the development of the FBOSS agent for programming ASICs and the Wedge switch specification's contribution to the Open Compute Project (OCP), aiming to foster open networking hardware and software.

2/17/2015

This post recaps the Networking @Scale conference, highlighting discussions on data center hardware and fabric, cloud networking, CDNs, low-latency trading networks, and application protocols. Key technical contributions include Google's OpenConfig for network configuration, JPMorgan Chase's low-latency trading network architecture, Facebook's '6-pack' open hardware modular switch and data center fabric, Fastly's CDN scaling strategies, Microsoft's synchronous geo-replication over Azure Tables, Netflix's global CDN scaling lessons, and Facebook's work on mobile networking challenges.

2/11/2015

This post introduces '6-pack,' an open hardware modular switch platform designed to address the limitations of traditional networking technologies at Facebook's scale. It builds upon previous work like the Wedge modular switch and FBOSS operating system, creating a disaggregated network architecture with independent switching elements. The 6-pack platform aims to provide flexibility, efficiency, and scalability by allowing any size switch to be built from common building blocks and is intended for contribution to the Open Compute Project.

11/14/2014

This post introduces the 'data center fabric,' a next-generation network architecture for Facebook's data centers. It details the shift from a cluster-based architecture to a disaggregated 'pod' system, aiming to create a single, high-performance network within the entire data center. Key innovations include the use of standard BGP4 with centralized override, a layer3 all-fabric design, and a modular topology with spine switches and pods to handle exponential growth in machine-to-machine traffic and enable rapid scaling.

11/14/2014

This post details the discovery and resolution of a metastable failure state in Facebook's network infrastructure related to aggregated links. The issue, which persisted for over two years, involved TCP traffic being imbalanced across aggregated links due to a hash-based routing strategy. The root cause was identified as an unintended consequence of a custom MySQL connection pool's Most Recently Used (MRU) policy, which, during query bursts, would preferentially select and keep congested links active. The fix involved switching to a Least Recently Used (LRU) connection pool with a maximum connection age, which stabilized network traffic and eliminated the metastable failure.

11/6/2014

Introduced 'year class', a classification system for Android devices based on hardware specifications (RAM, CPU cores, clock speed) to group similarly capable devices. This system allows for better segmentation of device performance and usage patterns, moving beyond OS version segmentation. It enables teams to make informed decisions about feature rollouts, performance optimizations, and content delivery based on device capabilities. Additionally, the post discusses the development of a 'connection quality' score to dynamically adjust app features based on actual network performance, independent of reported network type, to improve user experience on diverse network conditions.

11/5/2014

Introduced Proxygen, a C++ HTTP framework with server and client code, supporting HTTP/1.1, SPDY/3, SPDY/3.1, and developing HTTP/2 support. Proxygen was built for deep integration with Facebook infrastructure, code reuse across internal systems, scaling challenges with existing stacks, and implementing specific features like SPDY, WebSockets, and custom load-scheduling algorithms. The architecture is based on a four-part abstraction: session, codec, transaction, and handler, leveraging Thrift and Folly. Benchmarks of the echo server demonstrate parsing efficiency for binary protocols. Proxygen enabled rapid experimentation with HPACK on SPDY and quick adoption of evolving internal configuration systems.

6/19/2014

This post details significant improvements made to the Facebook Android app focusing on performance, data efficiency, networking, and application size. Key contributions include: deferring non-critical initializations to improve startup times, fetching News Feed stories earlier for faster loading on poor networks, optimizing image downloads using WebP compression (25-35% savings over JPG) and adaptive resolution loading based on viewport size, and fine-tuning image caching strategies. The networking stack was enhanced by adopting OkHttp, enabling faster retries and SPDY protocol support, leading to a 90% reduction in slow/failed image loads. Application size was reduced by 65% by utilizing Google Play's ability to upload multiple APKs for different device configurations. Automated tests and instrumentation were developed to monitor and prevent regressions in data usage and app size.

11/12/2013

This post introduces 'The Mature Optimization Handbook,' which provides a framework for performance measurement and optimization. It emphasizes the importance of data-driven decisions, avoiding premature optimization, and focusing on the critical 3% of code where optimization yields the most significant gains. It also highlights that performance optimization is a cost/benefit decision, best applied to mature systems, and can be a crucial feature when resources are limited or post-ship changes are difficult.

6/7/2013

This post details Facebook's adoption and internal implementation of IPv6, marking a significant step in supporting the growing number of internet-connected devices. It highlights the technical challenges faced, such as ensuring software compatibility with both IPv4 and IPv6, handling IP address representation and mathematics, and the ongoing work to make IPv6 support ubiquitous across their network. The post also notes the increase in IPv6 user adoption on Facebook from under 1% to over 4% in the year following World IPv6 Launch.

5/28/2013

Rewrote the Activity social plugin from the ground up to improve page-loading performance. Achieved a 5x reduction in payload size (257K to 48K) and reduced HTTP requests from 18 to 5. The plugin now loads nearly 2x faster (0.6s vs 1.14s) with initial painting starting in half the time (0.38s vs 0.68s). Optimizations included rewriting and reducing JavaScript/CSS sizes, untangling dependencies, improving packaging, and inlining all CSS.

4/29/2013

This post details the development of a natural language interface for Graph Search, which leverages a structured graph database. The system employs entity recognition, lexical analysis, and semantic parsing with a weighted context-free grammar to understand user intent. It handles variations in user input, including grammatical errors and synonyms, and uses a cost structure for ranking interpretations. The underlying infrastructure for Graph Search, including indexing and ranking, has been previously described.

2/26/2013

Rewrote the Like Box plugin from the ground up to significantly improve performance. Achieved 2-4x speed improvements by reducing component size (HTML/CSS/JS), inlining small CSS to prevent render-blocking, consolidating JavaScript into a single file loaded asynchronously, and optimizing for perceived performance by prioritizing faster initial browser paint.

6/6/2012

This post details the implementation of Layer 3 DSR with IP-in-IP tunneling for IPv4 and IP6-in-IP6 tunneling for IPv6 to support World IPv6 Launch. It describes the challenges of moving beyond single Layer 2 domains to Layer 3 clusters and the specific kernel patch developed to enable IPv6 tunneling on fallback devices. The post highlights the importance of IPv6 for future internet scalability and the successful integration of IPv6 into Facebook's infrastructure.

3/9/2012

Introduced a universal location tagging system for all content types, moving beyond mobile-specific check-ins. Developed a global places directory with a new indexing strategy (multi-polygon and prefix-based) for efficient global search and relevance ranking. Designed and implemented the timeline map feature to visualize location-tagged content. Created aggregation algorithms for map pins and systems for suggesting locations for existing albums. Launched location APIs for third-party applications to add and read location data.

12/20/2011

Introduced Doppler, a system for measuring worldwide packet latency using DNS hacks and JavaScript. Discussed the goal of achieving "0 degrees Kelvin" in profiling tools for precise system understanding and optimization. Highlighted the engineer's work in developing measurement tools, optimizing code, and collaborating with authors.

7/20/2011

Introduced Doppler, an internal project for mapping and measuring the network between users and Facebook. Doppler uses a lightweight JavaScript probe appended to a sample of page hits to perform continuous, large-scale network experiments (50 million per day). It measures end-to-end DNS latency by timing unique hostname requests and packet RTT by timing cached requests. The collected data is geolocated and rolled up by country and autonomous system number to identify network issues. Doppler is also used to evaluate TCP optimizations, hardware, and measure effective bandwidth. Future work includes mobile support, SSL, and IPv6, and using Doppler data for DNS-based global server load balancing.

6/9/2011

Meta successfully enabled IPv6 on their site for 24 hours as part of World IPv6 Day, serving over 1 million users over IPv6 with minimal impact on user experience. This success led to the decision to keep the Developer site dual-stacked and continue adapting the entire codebase and tools to support IPv6.

6/6/2011

Facebook participated in World IPv6 Day to test its infrastructure's readiness for IPv6, the next-generation internet addressing system. This involved testing their systems and code under IPv6 to identify and address potential issues, ensuring minimal disruption to users. The company had conducted prior studies and tests to gain confidence in their site's IPv6 readiness.

3/23/2011

This post introduces Facebook's founding membership in the Open Networking Foundation (ONF) and its commitment to Software-Defined Networking (SDN) and the OpenFlow protocol. It details how OpenFlow enables direct software control of network resources, leading to benefits such as automatic failure detection and rerouting, dynamic bandwidth provisioning via API, and energy savings through traffic-responsive power management. The post also explains how SDN and OpenFlow improve the utilization of fat tree and Clos network topologies for efficient server-to-server communication, crucial for scaled-out architectures like Facebook's.

1/12/2011

Facebook announced its participation in World IPv6 Day, a 24-hour global test of the IPv6 protocol. The post explains the necessity of IPv6 due to IPv4 address exhaustion and the challenges of dual-stacking websites. It contrasts IPv6 with Large Scale NAT (LSN), advocating for IPv6 as a more reliable and long-term solution. Facebook's participation is intended to test their infrastructure and code, and to encourage broader industry adoption and improvement of IPv6 products.

10/12/2010

This post details the early adoption of HTML5 features to improve web application performance and user experience. Specifically, it highlights the use of the Geolocation API for the 'Places' feature, the History API to enhance AJAX loading and URL management, and experiments with Web Storage for caching. It also mentions plans to explore WebSockets for more stable real-time applications, indicating a proactive approach to leveraging new web standards for network efficiency.

7/19/2010

This post introduces the concept of 'Internet Cartography' by analyzing global internet latency through user-generated traffic data. It details a method using two small network requests to measure DNS lookup and TCP+HTTP round-trip times, highlighting the influence of historical infrastructure, geography, and economic ties on network performance. The post also discusses the open-sourcing of the Boomerang measurement framework to encourage community-driven network analysis.

6/4/2010

Introduced BigPipe, a dynamic web page serving system that decomposes web pages into pagelets and pipelines them through server and browser execution stages. This system allows for the parallel generation and rendering of pagelets, significantly improving the speed at which users see content by overlapping server generation time with browser rendering time. Implemented entirely in PHP and JavaScript, it does not require changes to existing web browsers or servers.

5/17/2010

Introduced the architecture for Facebook's typeahead search, detailing the flow from user input to rendered results. This includes browser caching for initial results, AJAX requests, an aggregator service that delegates queries to specialized leaf services (global and graph), result merging, data fetching from memcache/MySQL, and privacy validation. The post also discusses performance optimizations, UI experimentation via A/B testing, and the trilemma of performance, recall, and relevance in real-time search.

2/19/2010

Introduced Time-to-Interact (TTI) as a key metric for measuring user-perceived performance, breaking it down into network time, generation time, and render time. Developed a component library for HTML and CSS to reduce bytes and improve reusability. Created the Primer JavaScript library to decrease JavaScript bytes and enable faster feature loading. Introduced the BigPipe system for pipelined generation and rendering of Pagelets, overlapping server generation with browser rendering to reduce overall Time-to-Interact.

8/28/2009

This post details an engineering intern's summer project focused on analyzing the impact of site speed and various UI loading strategies on user behavior. Experiments were conducted to measure the effects of site slowdowns on session time and page views, comparing immediate rendering with delayed JavaScript execution, and evaluating the benefits of scroll loading for News Feed content. The research demonstrated that improving site speed allows users to explore more content, that blank pages before interaction lead to lower usage, and that scroll loading significantly increases engagement, particularly for casual users. These findings directly informed the decision to increase the number of stories loaded via scroll loading.

4/8/2009

This post details the technical strategies employed to scale Facebook's live feed for the inauguration, handling over a million concurrent users. It describes optimizations for dynamically generated 'Friends' tab requests by reducing PHP initialization when no new statuses were returned, and for 'Everyone' tab requests by implementing multi-layered caching (memcache and CDN with short invalidation intervals). It also highlights challenges encountered with CDN caching, such as infinite loops due to stale data and users falling behind real-time updates, and the extensive load testing performed using a 'dark launch' approach to ensure system stability under peak load.

8/20/2008

This post details the technical challenges and solutions for scaling Facebook's infrastructure by opening a new datacenter on the east coast. Key contributions include: 1. Addressing latency by distributing data centers. 2. Solving cache consistency issues in a multi-datacenter setup with replication lag by modifying MySQL's replication stream to include memcache keys, enabling the secondary database to invalidate relevant cache entries. 3. Implementing traffic routing logic using Layer 7 load balancers and browser cookies to direct write operations to the primary datacenter and manage user experience during replication lag.