Internet Latency and Network Performance Analysis

Oct 2026NTS: Authenticated Time at MetaThis post details Meta's implementation and open-sourcing of Network Time Security (NTS) for its public time service. It explains the security vulnerabilities of unauthenticated NTP, particularly in the context of modern certificate validation and token expiry requirements. The post describes the two-phase NTS process (NTS-KE and Authenticated NTP), Meta's stateless cookie derivation mechanism using HKDF-SHA256 and epoch day, and the use of Chrony for client configuration. It highlights how NTS prevents man-in-the-middle attacks that could manipulate system clocks to validate expired certificates or extend credential lifetimes, and discusses limitations such as delay attacks and bootstrapping. The open-sourcing of the protocol, server, and client library is a key contribution.

Sep 2026Inside Petal: Building the World’s First Petabit-Class Transoceanic Subsea CableThis post details the engineering behind Petal, Meta's first petabit-class transoceanic subsea cable. It highlights the transition to multi-core fiber technology (2-core fiber) to achieve 1 Pbps capacity, doubling previous generations. Key technical challenges and solutions discussed include minimizing crosstalk between cores in the fiber and the design of a single-body 96-amp repeater using a Fan-In/Fan-Out (FIFO) interface to manage amplification for 2-core fiber. The post also touches on the manufacturing advancements required from partners like NEC and Sumitomo Electric Industries to produce this next-generation infrastructure.

Sep 2026ZGateway: Learnings from Putting a Proxy in Front of ZippyDBThis post introduces ZGateway, a new proxy layer designed to manage traffic for ZippyDB, Meta's widely-used key-value store. ZGateway addresses the challenges of a sprawling ZippyDB client fleet by acting as a unified entry point. Key technical contributions include: 1. **Connection Management:** ZGateway collapses the many-to-many mesh of TLS connections between clients and database hosts into bounded connections at the proxy tier, significantly reducing resource consumption and improving reliability by containing reconnection storms. 2. **Request Batching and Coalescing:** ZGateway groups requests from multiple clients destined for the same shard into single backend RPCs, amortizing fixed overheads, reducing QPS, and smoothing load. It also coalesces requests for hot keys, preventing stampedes against replicas. 3. **Traffic Routing and Migration:** ZGateway enables incremental and reversible migration of traffic via client-side configuration flags, allowing for controlled rollouts and rollbacks. 4. **Tenant Isolation and Admission Control:** The proxy tier enforces per-tenant admission control, validation, and shaping, preventing misbehaving tenants from impacting others. 5. **Read-Through Cache:** A caching tier is integrated to serve read requests directly, further improving performance. The post details the arithmetic behind fan-in/fan-out reduction and the safety mechanisms for in-memory batching (idle eviction, in-flight caps).

Aug 2026MetaRoCE: A New RDMA Transport Built for AI-Scale EthernetThis post introduces MetaRoCE, a new RDMA transport protocol designed from the ground up for AI workloads on commodity Ethernet at million-GPU scale. Key technical contributions include native out-of-order delivery, native multipathing with packet-by-packet spraying across first-class paths, loss tolerance by design (eliminating PFC and pause frames), a combined sender-driven ECN-based and receiver-driven fair-share rate hint congestion control, topology independence by relying only on ECN marking and ECMP, and unified connections at scale by separating ordered streams from bandwidth. The post details practical implementation on AMD Pensando NICs, showing higher throughput and lower flow completion times than RoCEv2, graceful degradation under packet loss, and autonomous recovery from plane failures. The protocol specification, reference software implementation, and compliance test suite are being released through the Open Compute Project (OCP).

Jul 2026Modernizing the Meta Ads Service With an Open-Source Kernel SchedulerThis post details the implementation and impact of a custom BPF-based scheduling policy using sched_ext to address latency regressions introduced by a Linux kernel upgrade (v6.6 EEVDF scheduler). The custom policy soft-partitions CPUs into latency-critical and less sensitive pools, improving L3 cache locality and reducing DRAM access. This resulted in a 28% reduction in ads retrieval tail latency, 3.28 MW power savings, and a 1.1% increase in weighted-ads-ranked. Subsequent user-space policy updates further reduced p99 latency by 60% and timeout errors by 18%. The post highlights sched_ext as a strategic asset for independent, rapid scheduler optimization.

Jun 2026Adopting AV1 for Real-Time Communication (RTC) at ScaleThis post details Meta's multi-year effort to adopt the AV1 video codec for real-time communication (RTC) across Messenger and WhatsApp. It outlines the technical and operational challenges encountered, including codec selection, device eligibility, rate control, and error resilience. Key contributions include the development of a low-complexity AV1 encoder with an ultra-low-complexity preset comparable to H.264/AVC, the selection of the dav1d decoder for its power efficiency, strategies for reducing binary size impact, and the creation of an ML-based device eligibility framework to reliably categorize device capabilities for AV1 support. The post also highlights AV1's advantages for screen content encoding using palette mode and intra-block copy.

Apr 2026Escaping the Fork: How Meta Modernized WebRTC Across 50+ Use CasesThis post details Meta's migration from a divergent WebRTC fork to a modular architecture based on the latest upstream version. Key contributions include the development of a dual-stack architecture with a shim layer for A/B testing, enabling runtime dispatch between legacy and latest WebRTC versions. This was achieved through automated C++ namespace renaming, handling of global symbols and macros, and a template-based helper library for version-specific logic. A code generation system was built to automate shim creation. The migration involved rewiring application references and using Buck build machinery to duplicate components for dual-stack builds, ultimately reducing binary size by 87% and enabling continuous upstream upgrades.

Mar 2026How Advanced Browsing Protection Works in MessengerThis post details the implementation of Advanced Browsing Protection (ABP) in Messenger, which uses Private Information Retrieval (PIR) and Confidential Computing (AMD SEV-SNP) to protect user privacy while analyzing potentially malicious URLs. It describes the challenges of URL prefix matching in a PIR setting, the development of a server-side ruleset generation process to balance bucket sizes and privacy, and the use of AMD SEV-SNP for secure processing of hash prefixes within a confidential virtual machine. The post highlights tradeoffs between privacy and efficiency in sharding strategies and the need for padding to obscure query lengths.

Feb 2026Building Prometheus: How Backend Aggregation Enables Gigawatt-Scale AI ClustersThis post introduces and details Meta's Backend Aggregation (BAG) technology, a new centralized Ethernet-based super spine network layer designed to interconnect multiple spine layer fabrics across data centers and regions for gigawatt-scale AI clusters like Prometheus. It describes how BAG connects thousands of GPUs across different network fabrics (DSF and NSF), detailing planar and spread connection topologies, inter-BAG connectivity, and the connection of BAG layers to L2 fabrics. It also covers hardware specifics (Jericho3 ASICs), routing (eBGP with UCMP), resilience design (port striping, failure domain analysis), and considerations for long cable distances requiring deep buffer switches.

Nov 2025Enhancing HDR on Instagram for iOS With Dolby VisionThis post details the implementation of Dolby Vision and ambient viewing environment (amve) metadata support within Instagram for iOS. It describes the challenges of preserving this metadata through Meta's video processing pipeline, which historically used FFmpeg lacking Dolby Vision support. Key technical contributions include: collaborating with Dolby and FFmpeg developers to implement Dolby Vision Profile 8.4 (HEVC) to Profile 10.4 (AV1) transcoding, enabling manual extraction of Dolby Vision metadata for AVSampleBufferDisplayLayer, and implementing client-side decompression for compressed Dolby Vision metadata. The post also highlights the A/B testing process to measure the impact of Dolby Vision on user engagement, identifying and mitigating a 100 kbps overhead issue through metadata compression, and ultimately demonstrating a positive impact on watch time.

Nov 2025Video Invisible Watermarking at ScaleThis post details Meta's development of a scalable invisible watermarking system for video content. Key technical contributions include: 1. Adapting ML-based watermarking for video by integrating with FFmpeg, overcoming GPU limitations related to video transcoding and inference latency. 2. Investigating and optimizing CPU-only inference, achieving performance within 5% of GPU performance with better operational efficiency and scalability through parallel FFmpeg processes. 3. Addressing optimization trade-offs between latency, bit-accuracy, visual quality, and BD-Rate, developing a novel frame-selection method to reduce BD-Rate impact. 4. Implementing custom post-processing and crowdsourced manual inspections to address visual quality regressions, as traditional metrics like VMAF and SSIM were insufficient.

Oct 2025Disaggregated Scheduled Fabric: Scaling Meta’s AI JourneyThis post introduces Disaggregated Scheduled Fabric (DSF), Meta's next-generation network fabric technology designed for AI training networks. It addresses challenges in traditional Clos-based networks by disaggregating line cards and fabric cards into distinct, interconnected hardware devices, creating a scalable, distributed system. DSF is a VOQ-based system powered by OCP-SAI and FBOSS, featuring packet spraying and a credit-based congestion control algorithm for near-optimal load balancing and lossless delivery. The post details the architecture for building massive clusters, including L1 zones (scaling units), L2 zones (dual-stage fabric), and L3 regions (super-spine interconnectivity), enabling clusters of up to 18K x 800G GPUs. It also introduces 'Input Balanced Mode' to manage traffic and prevent congestion during link failures by ensuring devices have equal or less input bandwidth compared to output bandwidth.

Oct 202510X Backbone: How Meta Is Scaling Backbone Connectivity for AIThis post details Meta's '10X Backbone' initiative to scale its network connectivity for AI workloads. It introduces three key techniques applied to the Express Backbone (EBB): DC Metro Architecture for simplified and standardized fiber deployment, IP Platform Scaling (scaling up with larger chassis and faster interfaces like 800G, and scaling out by adding planes or devices per plane), and IP/Optical Integration using ZR technology to reduce power consumption and improve rack density. It also outlines solutions for AI cluster expansion, including FR, LR, and ZR plugs with DWDM for varying distances.

Oct 2025OCP Summit 2025: The Open Future of Networking Hardware for AIThis post details Meta's advancements in networking hardware for AI training clusters, presented at OCP Summit 2025. Key contributions include the evolution of Disaggregated Scheduled Fabric (DSF) to a dual-stage architecture supporting up to 18,432 XPUs for scale-out interconnect, and the development of a new Non-Scheduled Fabric (NSF) architecture based on shallow-buffer Ethernet switches for large AI clusters. The post also announces the Minipack3N, a new 51 Tbps OCP switch based on NVIDIA's Spectrum-4 ASIC, and the introduction of 2x400G FR4 LITE and 400G/2x400G DR4 optics for 400G/800G optical interconnections. Meta is also a founding participant in the Ethernet for Scale-Up Networking (ESUN) initiative within OCP, contributing to the advancement of Ethernet for AI accelerator architectures.

Sep 2025Video Streaming With the AV1 Video Codec in Mobile DevicesThis post details Meta's collaboration with Vodafone and Google on a white paper recommending the adoption of the AV1 video codec for mobile devices. It highlights AV1's ability to enhance video compression by 30% over H.264 and VP9, leading to improved video quality on low/mid-tier handsets, reduced network capacity usage, and optimized computing power and storage. The post emphasizes the need for SoC vendors to adopt AV1 hardware or software decoders to ensure a buffer-free video experience for a wider user base. Meta has already implemented AV1-compatible codecs in its technologies.

Feb 2025How Precision Time Protocol handles leap secondsThis post details Meta's approach to handling leap seconds within the Precision Time Protocol (PTP) ecosystem. It introduces a 'self-smearing' algorithm for the fbclock library that shifts time by 1 nanosecond every 62.5 microseconds, providing a stateless and reproducible method for PTP synchronization. The post also discusses the tradeoffs of this approach, such as potential timestamp mismatches with NTP systems that use quadratic smearing, and highlights the benefits of using Coordinated Universal Time (UTC) over International Atomic Time (TAI) to avoid leap second complexities. It advocates for the industry-wide freeze of new leap seconds after 2035 to simplify timekeeping infrastructure.

Jan 2025ILA Evo: Meta’s journey to reimagine fiber optic in-line amplifier sitesThis post details Meta's ILA Evo effort to reimagine fiber optic in-line amplifier (ILA) sites. It describes the historical design constraints of ILAs and introduces new requirements for ILA Evo, including rapid deployment (3-4 days), reduced need for specialized heavy equipment, minimized concrete usage, and a target PUE of less than 1.5. The post outlines specific engineering approaches for building systems (lightweight FRP/GRP panels), foundations (low/no-concrete designs like screw piles), ISP installation (Manufactured Owner-Furnished Equipment - MOFE), cooling (advanced passive/liquid-based cooling, higher temperature set points), and backup power (alternatives to diesel generators like H2 fuel cells). It also discusses the macro benefits of ILA Evo, such as supply chain flexibility and stockpiling capabilities.

Dec 2024How we think about Threads’ iOS performanceThis post details how the Threads iOS team approaches performance monitoring and optimization. Key contributions include: defining and monitoring metrics like %FIRE (Frustrating Image-Render Experience) and TTNC (Time-to-Network Content) to track image loading and feed delivery speed, and cPSR (Creation-Publish Success Rate) for post reliability. It introduces SLATE, a custom 'Systemic LATEncy' logger, for consistent measurement of navigation latency across various user interactions and app states. The post also highlights the importance of keeping the app's binary size small, citing a 75% reduction for Threads compared to Instagram, and faster build times. A case study on adopting GraphQL for network requests is presented, emphasizing the need for performance validation using SLATE before full adoption. The team also conducted a 'boundary test' by injecting latency to understand user tolerance and impact on engagement.

Dec 2024Meta Andromeda: Supercharging Advantage+ automation with the next-gen personalized ads retrieval engineThis post introduces Andromeda, Meta's next-generation personalized ads retrieval engine. It details innovations in ML model architecture, feature representation, learning algorithms, indexing, and inference paradigms, specifically designed to leverage NVIDIA Grace Hopper Superchip and MTIA hardware. Key technical contributions include: a deep neural network custom-designed for superior performance on the Grace Hopper Superchip, achieving +6% recall improvement and +8% ads quality improvement; hierarchical indexing to support exponential ad creatives growth from Advantage+ creative, enabling efficient scaling to a large volume of ads creatives; and AI development efficiency through system simplification. Andromeda represents a new paradigm in ads retrieval by employing a state-of-the-art deep neural network with sublinear inference cost, hierarchical indexing jointly trained with retrieval models, model elasticity for agile resource allocation, and an optimized retrieval model with enhanced parallelism and low-latency, high-throughput GPU operators. The system achieves over 100x improvement in feature extraction latency and throughput compared to previous CPU-based components and enhances end-to-end model inference QPS by over 3x. Future work includes transitioning to an autoregressive loss function and integrating with MTIA and future GPUs for further scaling.

Oct 2024OCP Summit 2024: The open future of networking hardware for AIThis post details Meta's contributions to open networking hardware for AI training clusters, including two new disaggregated network fabrics (DSF) and a new NIC (FBNIC). It describes the architecture of DSF, which uses VoQ-based scheduling and OCP-SAI/FBOSS, and highlights specific hardware platforms like Arista 7700R4 series and Minipack3/Cisco 8501 switches. The FBNIC is presented as a multi-host foundational NIC with features like datapath isolation and OCP NIC 3.0 compliance. The post emphasizes the benefits of open hardware for flexibility, scalability, and efficiency in AI infrastructure.

Aug 2024How Meta animates AI-generated images at scaleThis post details optimizations for Meta AI's image animation feature, focusing on reducing inference latency and improving scalability. Key technical contributions include: halving floating-point precision from float32 to float16 (bfloat16) to reduce memory footprint and speed up operations; optimizing temporal-attention expansion by performing it after cross-attention's linear projection layers to reduce compute and memory; leveraging DPM-Solver with a linear-in-log signal-to-noise time to reduce sampling steps to 15; combining guidance and step distillation to reduce three forward passes per step to one, and distilling 32 teacher steps into eight student steps with one forward pass per step; and applying PyTorch optimizations like TorchScripting (continuous folding, fusing operations, reducing computational graph complexity) and freezing, later migrating to PyTorch 2.0 for more granular component optimization and multi-GPU inference. For large-scale deployment, a regional traffic management system was implemented to keep requests in the same region as their requester, minimizing network overhead. This system dynamically calculates routing tables based on load data, predefined thresholds, and routing rings to offload traffic to nearby regions when nearing capacity. To address dips in success rate due to GPUs being saturated, a probing system approximating retries was used to quickly check for free GPUs and prevent failures, with optimized retry settings for the router.

Aug 2024RoCE networks for distributed AI training at scaleThis post details Meta's evolution of RoCE networks for distributed AI training. Key contributions include: 1. **Topology Evolution**: Transition from simple star topologies to two-stage Clos topologies (AI Zones) for scalability, and the introduction of Aggregator Training Switches (ATSW) for DC-scale LLM training. It also discusses the oversubscription strategy and scheduler enhancements for cross-AI Zone traffic. 2. **Routing Evolution**: Moving from ECMP with low entropy issues to path-pinning, and then to Enhanced ECMP (E-ECMP) with Queue Pair (QP) scaling to increase flow entropy and improve performance for collectives like AllReduce. 3. **Congestion Control Shift**: Abandoning DCQCN for 400G deployments due to firmware issues and reduced visibility, and instead relying on PFC for flow control. For 400G+, a co-designed receiver-driven traffic admission mechanism between the collective library and RoCE transport is implemented to limit in-flight traffic and mitigate congestion.

Jun 2024MLow: Meta’s low bitrate audio codecThis post introduces MLow, Meta's new low-bitrate audio codec designed to improve audio quality for users on slow network connections and low-end devices. MLow achieves two-times-better quality than Opus at 6kbps (POLQA MOS 1.89 vs 3.9) while maintaining 10% lower computational complexity. It utilizes a split-band encoding approach (low and high frequencies) and builds on classic CELP concepts with advancements in excitation generation, parameter quantization, and coding schemes. MLow also enables more effective Forward Error Correction (FEC) strategies, improving audio quality in packet loss scenarios. The codec has been fully launched on Instagram and Messenger calls and is rolling out on WhatsApp.

May 2024Post-quantum readiness for TLS at MetaThis post details Meta's migration to post-quantum cryptography (PQC) for TLS, specifically focusing on hybrid key exchange using Kyber with X25519. It highlights the challenges encountered, including large packet sizes impacting TLS resumption and TCP Fast Open, leading to the use of Kyber512 for certain internal communications to maintain performance. A multithreading race condition in the liboqs library was identified and fixed. The post also discusses cross-domain resumption handshake thrash and the strategies employed to mitigate it. The technical contributions include the implementation of hybrid key exchange in Fizz, the use of liboqs, and the analysis and resolution of specific performance and stability issues related to PQC deployment at scale.

Mar 2024Threads has entered the fediverseThis post details the integration of Threads with the fediverse using the ActivityPub protocol. It describes the technical challenges and solutions for federating quote posts, including implementing FEP-e232 and _misskey_quote. It also outlines the phased approach to integration, focusing on server-to-server communication and managing interactions between federated and non-federated users within Threads.

Mar 2024Optimizing RTC bandwidth estimation with machine learningThis post details Meta's adoption of a machine learning (ML)-based approach to optimize Real-Time Communication (RTC) bandwidth estimation (BWE) and congestion control. It describes the transition from a parameter-tuned WebRTC module to an ML model that characterizes network types (random packet loss vs. congestion) and predicts future network conditions. The architecture combines LSTM for time-series data and dense layers for non-time-series data. The post shares experiment results demonstrating improvements in connection drop rate, video freeze percentage, and overall user experience across different network conditions, particularly for low-bandwidth users. It also outlines the challenges and learnings in generating training samples and deploying these models.

Mar 2024Better video for mobile RTC with AV1 and HDThis post details Meta's adoption and implementation of the AV1 video codec for real-time communication (RTC) across its apps (Messenger, Instagram, WhatsApp). It highlights the challenges of improving video quality for low-bandwidth users and the benefits of AV1 over older codecs like H.264. Key technical contributions include: custom video scalers that improve quality over default WebRTC scalers, region-of-interest encoding to prioritize important visual areas, and the use of AV1's royalty-free nature and software encoding capabilities for RTC scenarios where hardware encoders are not optimized. The post also discusses AV1's specific features beneficial for RTC, such as palette mode and intra-block copy for screen content, and reference picture resampling (RPR) to avoid costly key frames during resolution changes. It addresses the trade-offs of AV1's higher CPU/battery usage and outlines strategies for seamless codec negotiation and switching.

Feb 2024Simple Precision Time Protocol at MetaThis post introduces Simple Precision Time Protocol (SPTP), a simplified version of PTPv2 designed for Meta's data center deployments. SPTP reduces network communication and state machine complexity compared to standard PTPv2, leading to significant improvements in CPU, memory, and network utilization. It achieves this by reducing the number of packet exchanges to three (delay request, sync, followup/announce) and eliminating the need for persistent state on clients and servers. The post also details SPTP's reliability mechanisms, including multi-clock reliance and a clock ensemble approach using Chauvenet's criterion for outlier rejection and weighted averaging to ensure accurate timing information.

Nov 2023Watch: Meta’s engineers on building network infrastructure for AIThis post details Meta's network infrastructure evolution for AI workloads, including GenAI training and inference. It highlights the transition to GPU-based training with RoCE-based network fabrics and CLOS topologies. Specific contributions include RDMA deployment with RoCEv2, centralized traffic engineering for dynamic load balancing, and network observability tools (ROCET, PARAM, Chakra) for collective communication analysis. The Arcadia simulator is introduced for end-to-end AI system performance simulation, aiding data-driven decisions and future system design.

Sep 2023Using Chakra execution traces for benchmarking and network performance optimizationIntroduced Chakra execution traces, an open graph-based representation of AI/ML workload execution, to address limitations in traditional AI benchmarking (forecasting, cost, adaptability). Chakra captures compute, memory, and communication operations, dependencies, timing, and metadata. Developed tools for trace collection, analysis, generation, and simulation. Leveraged Chakra traces for benchmarking (Mystique, PARAM) and performance optimization (visualizing collective message sizes). Future plans include enhancing benchmarking with pre-execution traces and using generative AI to create representative traces for optimization and IP-protected sharing. Collaborating with MLCommons to drive industry adoption.

Aug 2023Using short-lived certificates to protect TLS secretsThis post details the implementation of Short-Lived Certificates (SLCs) for TLS private keys on Meta's edge networks. It describes the evolution of their TLS infrastructure from long-lived certificates to daily rotation, addressing challenges like certificate revocation, remote offload latency, and the need for robust automation. Key technical contributions include the ConfigBuilder for reliable certificate issuance and distribution, a prev-current-next model for rotation to handle synchronization delays, and a staged rollout process for validation. The system manages certificate issuance with a few months' lifespan but distributes them 10 days before expiration, and rotates them daily, limiting private key exposure to 10 days. The post also discusses scaling challenges, such as overloading secret stores during rotation, and the solutions implemented, including caching and staggered secret fetches.

Jul 2023Bringing HDR video to ReelsThis post details the technical implementation of bringing High Dynamic Range (HDR) video support to Meta's Reels feature. It addresses the challenges of diverse device capabilities by implementing client-side tone mapping (using native APIs on iOS and custom shaders on Android) to convert HDR to SDR for compatibility, and server-side tone mapping to generate both HDR and SDR renditions for optimal playback across devices. It also discusses managing brightness consistency between SDR UI elements and HDR video content. The Meta Scalable Video Processor is leveraged for efficient HDR processing.

Apr 2023A fine-grained network traffic analysis with MillisamplerIntroduced Millisampler, a lightweight, fine-grained network traffic characterization tool. Millisampler uses an eBPF-based tc filter to collect time series data on ingress/egress traffic volumes, active flows, ECN marks, and retransmissions at configurable timescales (100us, 1ms, 10ms). It employs per-CPU variables to avoid locks and samples periodically to minimize overhead, storing data locally for on-demand analysis. Millisampler has been instrumental in debugging network performance issues, such as synchronized traffic bursts and NIC driver bugs, and in characterizing traffic patterns for service performance optimization.

Apr 2023Why xHE-AAC is being embraced at MetaThis post details Meta's adoption and deployment of the xHE-AAC audio codec on Facebook and Instagram for features like Reels and Stories. It explains the benefits of xHE-AAC, including integrated loudness management via metadata for consistent audio levels and adaptive bit rate audio using immediate playout frames (IPFs) for seamless switching between audio qualities based on network conditions. The deployment process involves using the Fraunhofer IIS encoder SDK and shaka-packager for DASH streaming, with a two-pass encoding mode for loudness measurement and LRAC DRC. The post also highlights the resulting improvements in user experience, such as reduced playback stalls.

Feb 2023How Meta brought AV1 to ReelsThis post details Meta's successful integration and deployment of the AV1 video codec for Facebook and Instagram Reels. It outlines the technical rationale for choosing AV1, highlighting its superior compression efficiency (approx. 30% better than VP9/HEVC) and royalty-free nature. The post delves into the optimization of AV1 software encoders (specifically SVT-AV1) and decoders (dav1d), benchmarking their performance against other codecs and demonstrating significant improvements in quality-to-bitrate ratios. It also describes the advanced Adaptive Bit Rate (ABR) encoding strategy using convex hull optimization and a two-stage hybrid hardware/software approach to balance encoding speed and quality. The challenges and solutions for client-side decoding, particularly on mobile devices without hardware AV1 support, are also discussed.

Feb 2023Improving Meta’s global mapsThis post details significant upgrades to Meta's global basemap system over the past two years. Key technical contributions include the launch of raster-first basemaps alongside vector maps for improved rendering performance across devices, the integration of the Daylight distribution from OpenStreetMap and the creation of Earth Tables for simplified OSM tag handling, and the introduction of AI-based translations for map labels. Cartographic design improvements include a softened color palette, deprioritization of roads to emphasize community spaces, and the addition of detailed pavement data and iconography for points of interest. The system is also evolving to incorporate data from the Overture Maps Foundation and explore 3D geospatial data.

Jan 2023Watch Meta’s engineers discuss optimizing large-scale networksThis post details Meta's engineers discussing optimizations for large-scale networks across three themes: data center networking (FBOSS, SAI, in-house switches), WAN optimizations (global optical network, submarine fiber optics like Bifrost, Echo, 2Africa), and operational efficiency (Millisampler for fine-grained traffic analysis, Network SLOs for QoE correlation, and Shiv for L4 routing consistency).

Nov 2022PTP: Timing accuracy and precision for the future of computingThis post details Meta's deployment of Precision Time Protocol (PTP) across its data centers to achieve nanosecond-level timing accuracy and precision, surpassing the millisecond-level capabilities of NTP. It explains the technical challenges of network latency and clock synchronization, highlighting PTP's use of hardware timestamping and transparent clocks. The post also describes the development of custom hardware, the Time Appliance and Time Card, to support PTP at hyperscale, and the open-sourcing of these components through the Open Compute Project. The benefits of PTP for current products (messaging, gaming, video conferencing) and future technologies (metaverse, AI synchronization) are emphasized.

Nov 2022How Precision Time Protocol is being deployed at MetaThis post details Meta's migration to Precision Time Protocol (PTP) for nanosecond-level system synchronization, a significant upgrade from the millisecond precision of NTP. It covers the architectural components of the PTP deployment, including the GNSS antenna, PTP rack with Time Appliances (featuring Time Cards and Network Cards with hardware timestamping), and the PTP client. Innovations include the use of GNSS-over-fiber technology for signal distribution, diversification of Time Card hardware through OCP collaboration, and the development of 'ptp4u' to scale PTP server capacity to 1 million clients per appliance. The post also highlights the importance of precise timing for distributed system guarantees like linearizability and showcases performance improvements over NTP.

Sep 2022Launching a new Chromium-based WebView for AndroidThis post details the development and rollout of a custom Chromium-based WebView for the Facebook app on Android. This custom WebView acts as a drop-in replacement for the System WebView, allowing it to update in sync with Facebook app updates. This approach addresses security risks associated with outdated System WebViews by enabling timely application of Chromium security patches. It also resolves stability issues by preventing app crashes that occur when the System WebView is updated independently. Furthermore, the custom WebView improves rendering performance by enabling a separate GPU process for its compositor, which is not possible with the standard System WebView due to Android's synchronous widget layout requirements. The post also highlights Meta's ongoing contributions to upstream Chromium and web standards.

Sep 2022Network Entitlement: A contract-based network sharing solutionIntroduced Network Entitlement, a contract-based network sharing solution to manage scarce network resources. This framework allows services to reserve bandwidth per region per class of service (CoS) with an agreed SLO guarantee. Key properties include isolation for reliability, guarantee and accountability, abstraction of network complexity, observability, and a work-conserving approach. The solution consists of contract abstraction using the hose model, a dynamic SLO-based guarantee service that prioritizes high-touch services, a distributed runtime enforcement system at end hosts, and policy verification through production drills.

Jul 2022It’s time to leave the leap second in the pastThis post details Meta's position and technical approach to the leap second problem, advocating for its elimination. It explains the historical context, the technical challenges posed by leap seconds (including positive and negative adjustments), and the complexities of 'smearing' the leap second adjustment across distributed systems. Meta's specific smearing strategy involves a 17-hour window using quadratic algorithms, highlighting the non-standardized nature of this practice across the industry and the operational risks associated with NTP server restarts during smearing. The post also cites industry examples of outages caused by leap seconds, reinforcing the argument for their removal.

Jul 2022Watch Meta’s engineers discuss QUIC and TCP innovations for our networkThis post details several innovations in QUIC and TCP for Meta's network. It introduces Quick cache DSR for bypassing intracluster communication in CDNs, QUIC Jump Start for reducing backbone network transfer startup delays, DCTCP for sustained data center congestion, receiver window tuning for data center bursts, NetEdit for fine-grained network tuning at scale using BPF, and a network entitlement framework for WAN resource sharing. These contributions focus on improving efficiency, reliability, and performance across various network layers.

Jun 2022Under the hood: Meta’s cloud gaming infrastructureThis post details Meta's cloud gaming infrastructure, focusing on edge computing, GPU virtualization, and video/audio streaming. It highlights the use of edge sites for low latency, NVIDIA GPUs for game rendering, and Twine for cluster management. Containerization is used for Windows and Android games. WebRTC with SRTP is employed for streaming, with improvements made through GPU encoding and hardware decoding. Security measures include isolation and threat modeling. The infrastructure is presented as foundational for future metaverse development, emphasizing the need for ultra-low latency, high scalability, and robust networking.

Nov 2021OCP Summit 2021: Open networking hardware lays the groundwork for the metaverseThis post details Meta's advancements in open networking hardware for data centers, including the introduction of Wedge 400/400C TOR switches utilizing Broadcom and Cisco ASICs, offering significantly increased switching capacity and performance. It also highlights the migration of FBOSS to OCP SAI, enabling broader silicon vendor support and easier onboarding of new hardware. Furthermore, it introduces next-generation 200G/400G fabric switches like Minipack2 and Arista 7388X5, emphasizing reduced power consumption and compatibility with future upgrades. These developments are framed within the context of building infrastructure for the metaverse and promoting open, disaggregated networking.

Oct 2021RTMP Go Away: Lossless reconnections for live streamingIntroduced the RTMP Go Away mechanism, a new feature for the Real Time Messaging Protocol (RTMP) that allows live servers to signal an impending connection termination to clients. This enables clients to reconnect at a logical media boundary (e.g., next keyframe) with zero data loss, reducing reconnection times from up to 20 seconds to minimal interruption. The contribution includes spec updates to the RTMP connect packet and a new protocol message type for the go away signal, along with a reference implementation in FFmpeg.

Oct 2021More details about the October 4 outageThis post details a major outage caused by a command issued during routine maintenance of Meta's global backbone network capacity. The command unintentionally took down all backbone connections, effectively disconnecting Facebook data centers globally due to a bug in the audit tool. This led to a secondary issue where DNS servers withdrew BGP advertisements, making them unreachable. The recovery process was complicated by the inaccessibility of data centers and the failure of internal DNS-dependent tools. Onsite engineers were dispatched to debug and restart systems, a process that took time due to physical and system security measures. The company leveraged "storm" drills to manage the load surge when bringing services back online. The post emphasizes learning from the incident, strengthening testing and drills, and acknowledges the trade-off between security hardening and recovery speed.

Oct 2021Update about the October 4th outageThis post details a major outage caused by a faulty configuration change on backbone routers coordinating network traffic between data centers. The disruption cascaded, halting services. The root cause was identified as a configuration error, not malicious activity, and no user data was compromised. The company is working to understand the incident to improve infrastructure resilience.

Aug 2021Open-sourcing a more precise time applianceThis post details the development and open-sourcing of the Open Compute Time Appliance and the associated Time Card. The Time Card is a PCIe card that integrates a GNSS receiver, a miniaturized atomic clock (MAC), and an FPGA to provide nanosecond-accurate time synchronization. The system utilizes a GPS disciplined oscillator (GPSDO) and a NIC with hardware time stamping to synchronize the physical hardware clock. The Time Card's time engine interpolates time between PPS signals and uses a PI control loop for calibration. The open-source nature of this project aims to provide a more accessible and customizable solution for accurate timekeeping compared to proprietary off-the-shelf devices.

Aug 2021Risk-driven backbone management during COVID-19 and beyondThis post details Meta's risk-driven backbone management strategy implemented during the COVID-19 pandemic. It introduces a simulation system that quantifies network risk using metrics for demand loss, availability, and latency stretch, based on failure scenarios and component characteristics (MTBF, MTTR). This system guided proactive capacity deployment and traffic priority management (downgrading QoS for latency-insensitive traffic) to maintain service SLOs under unprecedented stress. The research, presented at NSDI 2021, aims to help operators build more resilient networks.

Jun 2021Network hose: Managing uncertain network demand with model simplicityIntroduced the 'network hose' planning model to manage uncertain network demand. This model shifts from forecasting pairwise traffic matrices between data centers to forecasting aggregate ingress and egress traffic per data center, reducing forecast complexity by an order of magnitude. To address the continuous demand sets within the hose constraint, a method was developed to identify critical reference demand sets by analyzing network cuts and data center locations, focusing on simple cuts that are likely to drive resource needs. This simplifies network design and operational workflows.

May 2021Peering automation at FacebookThis post details the development and implementation of an automated system for managing public peering at Facebook. Previously a manual, time-intensive, and error-prone process, the new system allows peers to request peering sessions via a web portal, leveraging PeeringDB OAuth for authentication. The system automates BGP session configuration on Facebook's routers and monitors for session establishment, sending automated reminders to peers. This has resulted in a significant reduction in manual effort (over 8 hours per week) and the automatic setup of over 1,400 public peering sessions. The post recommends leveraging PeeringDB OAuth as an industry standard for public peering automation and discusses future plans to automate private network interconnects (PNIs).

May 2021Running Border Gateway Protocol in large-scale data centersThis post details the scalable design, software implementation, and operations of Facebook's data center routing design based on Border Gateway Protocol (BGP). It introduces a uniform AS numbering scheme, hierarchical route summarization, and policy configurations for reliable communication. It also describes an in-house BGP agent with a multi-phase deployment pipeline for testing and incremental deployment, including unit testing, emulation, and canary testing.

Mar 2021Cutting Threads’ send latency in halfThis post details optimizations made to the server-side media ingestion endpoint for Instagram Threads to reduce send latency. The primary improvement involved identifying that the write-through cache's backing store was in a faster region, causing unavoidable cross-region round trips for writes. The solution was to route requests to the faster region via a special HTTP header and routing rule, effectively batching these writes and reducing the endpoint's execution time. This resulted in an average send latency reduction of 29%.

Dec 20202020 year in review: Connectivity innovations, faster apps, and progress toward net zeroThis post details the adoption and impact of QUIC and HTTP/3 on Meta's internet traffic, showing significant improvements in request errors, tail latency, and other metrics. It also highlights the complete rewrite of Messenger for iOS (Project LightSpeed) resulting in a smaller, faster app with fewer lines of code, and the redesign of the Facebook web app using React and Relay. Additionally, it covers the use of automation and ML for infrastructure hardware maintenance, new tools for engineers (Health Compass, Incident Tracker), and hardware optimization for microservices (SoftSKU, Accelerometer). Connectivity innovations include aerial fiber deployment robots, the 2Africa subsea cable, SuperCell for rural connectivity, and the Evenstar program for affordable broadband.

Dec 2020A smaller, faster video calling library for our appsIntroduced rsys, a new, from-scratch video calling library built on the latest WebRTC, resulting in a 20% reduction in binary size and cross-platform support (Android, iOS, macOS, Windows, Linux). Achieved size optimization through modular architecture, selective feature compilation using Bazel's `select`, a Flux-based framework for feature development, and leveraging smaller libraries like Boost.SML over heavily templated ones. Unified signaling and media stacks using state machines to reduce code duplication and improve maintainability. Abstracted OS-dependent functionality with generic interfaces and used `cxx_library` for platform-specific configurations.

Oct 2020How we scale Live streaming for millions of viewers simultaneouslyThis post details Meta's significant advancements in scaling live streaming infrastructure to support millions of concurrent viewers for both user-generated content and professional broadcasts. It highlights the development of new ingestion capabilities with built-in redundancy and failover mechanisms, the scaling of delivery infrastructure to combat the 'thundering herd problem' through request coalescing and cache sharding, and proactive collaboration with ISPs to manage network congestion. The post also outlines future challenges and explorations, such as using LTE-Broadcast for last-mile delivery and the need to optimize for a high volume of simultaneous smaller-reach streams during events like New Year's Eve.

Oct 2020How Facebook is bringing QUIC to billionsThis post details Meta's development and deployment of QUIC and HTTP/3, replacing TCP and HTTP/2 for over 75% of their internet traffic. Key contributions include the development of the mvfst QUIC implementation, integration with Proxygen Mobile and Fizz, and strategies for deploying QUIC across Facebook and Instagram apps and web. The post highlights performance improvements (e.g., 6% reduction in request errors, 20% tail latency reduction for dynamic requests; up to 22% MTBR improvement for video) and challenges encountered, such as app logic heuristics tuned for TCP causing regressions, and the need to optimize mvfst's CPU efficiency and the BBR congestion control for static content. Solutions involved performance testing tools, pacing UDP packets, using GSO, optimizing data structures, and tuning flow control parameters. The deployment strategy involved phased rollouts, starting with internal traffic and then expanding to public-facing applications.

Mar 2020Building a more accurate time service at Facebook scaleThis post details the evolution of Facebook's time synchronization service, moving from ntpd to chrony to achieve sub-millisecond precision. It describes the four-layer NTP service architecture, the comparison of ntpd and chrony's accuracy and resource consumption, and the validation of chrony's performance using 1PPS measurements in a timing laboratory. The post also announces the open-sourcing of Facebook's NTP libraries.

Nov 2019Networking @Scale Boston 2019 recapThis post details presentations from the Networking @Scale Boston 2019 conference, focusing on challenges and solutions for reliable networking at scale. Key contributions include discussions on network reliability evolution, building and automating global WANs, analyzing interdomain routing, Anycast content delivery architectures, enterprise Wi-Fi mesh networks (SOMA), security performance management, enforcing encryption at scale (TTLS), improving QUIC CPU performance, offloading connection encryption with SmartNICs, performance tools for Envoy scalability, adaptive cache networks, and building edge compute platforms for Stadia.

Nov 2019Groundbreaking event for our newest long-haul fiber routeThis post details a groundbreaking event for a new long-haul fiber route connecting data centers in Ohio, Virginia, and North Carolina. It emphasizes the strategic importance of this infrastructure for maintaining reliable, resilient, and scalable access to Meta's services for its global user base, highlighting a pragmatic approach to network infrastructure investment.

Nov 2019Evaluating COPA congestion control for improved video performanceThis post details the evaluation and implementation of COPA, a tunable delay-based congestion control algorithm, within Meta's QUIC implementation for Facebook Live video streaming on Android. It presents comparative analysis against CUBIC and BBR v1, demonstrating COPA's ability to significantly reduce video latency (application RTT) and improve goodput, particularly in worse network conditions. The post also investigates RTX overhead and potential network policing scenarios, highlighting COPA's proactive approach to congestion management by reducing CWND based on queuing delay rather than solely packet loss.

Nov 2019Delegated credentials: Improving the security of TLS certificatesThis post introduces and explains the concept of 'delegated credentials,' a new extension to the TLS protocol being developed with industry partners. It addresses the challenges of managing TLS certificates at scale, particularly the security risks associated with long certificate lifetimes and the limitations of traditional revocation mechanisms. Delegated credentials offer a way to significantly shorten certificate lifetimes (to hours or days) by allowing servers to sign their own short-lived credentials using their primary certificate, thereby improving security and reducing reliance on revocation without impacting reliability. The post also highlights Meta's implementation of this in Fizz and provides a test endpoint.

Sep 2019Networking @Scale 2019 recapThis post summarizes the Networking @Scale 2019 conference, highlighting discussions on reliable networking at scale. Key contributions include presentations on: secure reliability in infrastructure, design principles for network availability, Facebook's BGP++ deployment and outages, Microsoft's Open Network Emulator (ONE) for preventing network changes from becoming outages, Cloudflare's learnings from bootstrapping 1.1.1.1, operational challenges and solutions for Facebook's Express Backbone (EBB) SD-WAN, Netflix's client-side network measurement system (Probnik), Facebook's infrastructure for enforcing encryption at scale using BPF programs and Transparent TLS (TTLS), and AWS's SAFE protocol for incident response and recovery.

Jun 2019Facebook’s open source browser contributionsThis post details Meta's contributions to evolving the web platform to improve web application performance, specifically addressing the performance gap between web and native applications. Key contributions include the proposal and implementation of the `isInputPending()` API to improve JavaScript scheduling and responsiveness, the integration of retired instruction counts into Chrome's performance tracing system for better benchmarking, the proposal and implementation of a native JavaScript Self-Profiling API for collecting execution stack traces with low overhead, and the partnership with Mozilla to propose and implement the experimental BinAST binary encoding format for JavaScript to improve parsing efficiency by allowing selective parsing of code.

May 2019Building Facebook’s service encryption infrastructureThis post details Meta's migration of its internal data center service encryption from Kerberos to TLS. It highlights the challenges of managing Kerberos at scale, leading to the adoption of TLS with X.509 certificates for improved operability and performance. Key contributions include the design principles for managing the security-operability trade-off, the use of session tickets for efficient symmetric key cryptography, and the development of an internal certificate authority (ICA) for robust identity provisioning during container setup, ensuring end-to-end defense in depth for microservices.

May 2019Extending DHCPLB: The path from load balancer to serverThis post details the evolution of Meta's DHCP infrastructure, moving from ISC-DHCP to ISC Kea, and finally to an extended DHCPLB acting as a server. The primary technical contribution is the replacement of the single-threaded Kea DHCP server with a multithreaded DHCPLB server implementation. This addresses the Recv-Q buffer bottleneck and improves throughput by leveraging a new Go DHCP library. The post highlights performance gains, including handling the same traffic volume with 10x fewer servers and maintaining constant latency under high QPS. It also details the development of a new DHCPLB server mode that generates responses instead of forwarding, and the use of DHCPLB for A/B testing server implementations.

Apr 2019Faster input events with Facebook’s first browser API contributionThis post details the proposal and contribution of the `isInputPending` API to web browsers, specifically Chrome. The API addresses the trade-off between fast page loading and quick event response by allowing JavaScript to check for pending input events before continuing long-running tasks. This helps prevent user interactions from being queued for extended periods, improving overall web responsiveness. The post describes the problem of JavaScript queueing time, the proposed solution, its implementation in Chrome behind an origin trial, and its potential impact on web performance.

Mar 2019Reinventing Facebook’s data center networkThis post details the evolution of Meta's data center network architecture, introducing the F16 fabric design and HGRID aggregation layer. F16 offers 4x the capacity of previous designs with a simpler, flatter topology, achieved by using 100G CWDM4-OCP optics and single-chip 128-port fabric switches (Minipack) instead of multi-chip 400G designs. Minipack is a new modular building block switch that consumes 50% less power and space. HGRID is a larger aggregation layer designed to scale to six buildings per region, utilizing Minipack as its building block and flattening the regional network for East-West traffic. The post also highlights the continued use and evolution of FBOSS as the network operating system, supporting multiple generations of topologies and hardware platforms.

Dec 2018DNS over TLS: Encrypting DNS end-to-endThis post details a pilot project with Cloudflare to implement DNS over TLS (DoT) between Cloudflare's recursive resolvers and Meta's authoritative name servers. It analyzes the feasibility, latency overhead, and computing overhead of this protocol in a production environment. The pilot demonstrates that DoT can be implemented at scale without negatively impacting user experience, with p99 DNS latency between Cloudflare and Meta name servers being comparable to UDP baselines due to TLS session resumption. The post also highlights the importance of operator input in protocol development and encourages industry partners to join the pilot.

Oct 2018MobileLab: Highly accurate testing to prevent mobile performance regressionsIntroduced MobileLab, a new system for highly accurate mobile performance testing designed to prevent regressions as small as 1 percent. MobileLab achieves this by establishing a deterministic testing environment through several key technical innovations: locking CPU and GPU frequencies to fixed settings, using an HTTP proxy to simulate consistent network responses, managing application on-disk state across trials, remounting data directories to RAM disks for consistent disk I/O, resetting device clocks, dismissing crash dialogs, and replacing logcat tailing with direct socket communication for metric reporting. This system has improved confidence intervals by 7x and reduced false positives by 75 percent compared to previous methods.

Sep 2018FBOSS: Building switch software at scale and in the openThis post introduces FBOSS (Facebook open switching system), an open-source switch software designed for large-scale, rapidly growing networks. It details FBOSS's lean, modular architecture, which strips away unnecessary components and leverages an 'deploy early and iterate' approach for easier implementation, maintenance, and updates. The post highlights FBOSS's role in scaling Facebook's data center deployments by 30x over two years, enabling agile responses to new developments such as IPv6 implementation and the Fabric Aggregator. The authors share their experiences and challenges using FBOSS in the Facebook network, encouraging further research and development in open and streamlined infrastructure design.

Aug 2018Deploying TLS 1.3 at scale with Fizz, a performant open source TLS libraryThis post details the development and large-scale deployment of Fizz, a C++ TLS library designed to implement TLS 1.3. It highlights Fizz's performance improvements over TLS 1.2, including reduced latency (especially with 0-RTT) and CPU utilization. Key technical features discussed are asynchronous I/O by default, scatter/gather I/O for zero-copy operations, and secure state machine design to prevent vulnerabilities. The post also covers the challenges of deploying TLS 1.3 at scale, including middlebox interference, and the workarounds implemented in Fizz. The open-sourcing of Fizz is announced as a contribution to the broader internet community.

Jun 20182018 Networking @Scale recapThis post details the 2018 Networking @Scale conference, highlighting discussions on high-speed optical networking, zero-touch provisioning with Vending Machine, ML-driven network resilience modeling, at-scale optical network management, IPv6 adoption data, the HyperPlane system for network load balancing, Edge Fabric for egress traffic management, and the open-sourcing of Katran, a scalable L4 load balancer utilizing XDP and eBPF. It also touches on the application perspective of networking from Blizzard.

Jun 2018How IPv6 deployment is growing in U.S. and other countriesThis post details the growth of IPv6 deployment across the U.S. and other countries, highlighting its performance benefits over IPv4 and the critical need for migration due to IPv4 address exhaustion. It shares specific adoption rate data for U.S. mobile carriers and ISPs, noting T-Mobile's aggressive push to 100% and Comcast's 75% weekend deployment. The post also showcases exponential growth in countries like Taiwan and significant milestones in India, Mexico, Vietnam, and Thailand. A key technical detail is the adjustment of the Happy Eyeballs algorithm by increasing the timeout from 50ms to 150ms, which improved application responsiveness and increased IPv6 adoption by preventing unnecessary fallbacks to IPv4.

May 2018Open-sourcing Katran, a scalable network load balancerThis post introduces Katran, a new software-based network load balancer that leverages eXpress Data Path (XDP) and eBPF for improved performance and scalability. It details the challenges of serving requests at Facebook scale, the requirements for a high-performance load balancer, and the evolution from a first-generation L4LB based on IPVS to Katran. Katran's reengineered forwarding plane allows it to run on commodity Linux servers, coexist with other services, and offers improved instrumentation and debugging.

May 2018Scaling the Facebook backbone through Zero Touch ProvisioningThis post introduces Vending Machine, a new workflow system for automating the provisioning of Facebook's global backbone networks and edge network points-of-presence. It details the challenges of rapid growth in heterogeneous IP and optical networks, moving from manual MOPs to a software-centric approach. The system leverages Zero Touch Provisioning (ZTP) with custom Python agents executed on network devices for initial configuration and software upgrades. Vending Machine's core innovation is its workflow framework, which breaks down provisioning tasks into independent 'steps' that can be developed and executed in parallel, allowing for iterative automation of all manual MOP procedures. This system has enabled faster, more creative problem-solving and an iterative approach to network building and deployment tools.

Apr 2018Facebook video adds AV1 supportThis post details the integration of the AV1 video format into Facebook video playback. It describes the motivation for using AV1 (improved compression efficiency), the challenges (slow encoding speed), and the implementation strategy (segment-based encoding for parallel processing). It also outlines how users can view AV1 content using Chrome Canary with specific flags enabled. The post highlights the collaboration with the Alliance for Open Media (AOMedia) and the future plan to switch to the official AV1 release version once browser support is widespread.

Apr 2018AV1 beats x264 and libvpx-vp9 in practical use caseThis post details Meta's practical testing of the AV1 video compression codec against x264 and libvpx-vp9. It outlines the methodology, encoder configurations, and experimental results, demonstrating AV1's superior compression efficiency (30-50% better than alternatives) at the cost of significantly longer encoding times. The findings are presented with detailed metrics (BD-rate, PSNR, SSIM, encoding time ratios) and are intended to inform engineers about AV1's potential for real-world production environments.

Mar 2018Fabric Aggregator: A flexible solution to our traffic demandIntroduced the Fabric Aggregator, a distributed network system designed to scale network capacity between data center fabrics. This system leverages disaggregated building blocks like Wedge 100 switches and FBOSS, with a 2-layer cross-connect architecture for east/west and north/south traffic. It allows for independent scaling of capacities, interchangeable subswitches, and isolated failures for improved reliability. The post also details operational aspects for subswitch/node maintenance and backplane cabling configurations for single-rack and multi-rack deployments.

Mar 2018Performance @Scale 2018 recapThis post recaps the Performance @Scale 2018 conference, highlighting technical discussions and presentations on managing performance challenges at scale. Key technical contributions and discussions include: Andre Vachon (Microsoft) on Execution Graphs for distributed trace processing and performance regression detection in Azure VM deployments; Delyan Kratunov (Facebook) on understanding mobile app performance in the wild and the open-source release of Profilo; Yang Yang and Ritesh Maheshwari (LinkedIn) on ThirdEye, a generic anomaly detection platform for time series metrics, applied to real user monitoring data; Rico Mariani (Facebook) on iOS VM and loader considerations impacting application startup time; Brien Colwell (Headspin) on a new test infrastructure for automated mobile performance tests in real user conditions; Guy Cirino (Netflix) on building a Magic Modem for global network emulation; Xiaowei Jiang (Alibaba) on scaling real-time infrastructure for global shopping holidays; and Nick Gavalas (Facebook) on efficient static resource delivery mechanisms for Facebook on the web.

Mar 2018Profilo: Understanding app performance in the wildIntroduced Profilo, a high-throughput, mobile-first performance tracing library, now open-sourced for Android. Profilo enhances the ability to collect and analyze detailed telemetry from mobile apps in production, improving the diagnosis of performance regressions by providing precise root-cause analysis for issues as small as tens of milliseconds of CPU time. It enables new causality analyses and more precise understanding of metrics like 'scroll fluidity' and 'app responsiveness'. Key technical contributions include a custom Java stack unwinder that bypasses official Java APIs for efficiency and accuracy, and the capture of 'atrace' (systrace) telemetry in production to analyze VM and Android framework behavior, garbage collection, I/O operations on the UI thread, and lock contention.

Feb 2018How production engineers support global events on FacebookThis post details the production engineering strategies for supporting global events on Facebook, focusing on handling traffic spikes from high-bandwidth content like Facebook Live. It outlines three categories of load variance (routine, spontaneous, planned) and elaborates on the planning and architectural considerations for planned events like New Year's Eve. The post describes the architecture of Facebook Live, identifying network, CPU, and storage as key resource types affected by increased broadcasts. It highlights the importance of peak concurrent broadcasts as a critical metric for planning and discusses the challenges of accommodating this metric in real-time. The article also details the estimation of New Year's Eve traffic using historical data and growth projections, and describes scaling efforts including hardware allocation and efficiency improvements like consolidating media segments. Finally, it explains three load testing methods: artificially increasing load on tiers, synthetic load tests of dependencies, and shadow traffic.

Dec 20172017 Year in review: Better global networksThis post details advancements in Meta's global networking infrastructure in 2017. Key contributions include the deployment of 100Gbps networking with optimized 100G single-mode optical fiber, contributing designs to the Open Compute Project (OCP). The company split internal and external traffic into separate networks, building the Express Backbone for cross-data center traffic with a hybrid traffic engineering approach, and enhancing the Edge Fabric for steering user traffic from Points of Presence (PoPs). FCR, a network device health monitoring service, was also introduced. For global connectivity, efforts included expanding Express Wi-Fi hotspots, supporting Free Basics, and contributing to the Telecom Infra Project (TIP). Technological advancements included the second test flight of the Aquila solar-powered aircraft, record-breaking wireless data transfer rates (36 Gb/s with millimeter-wave, 80 Gb/s with optical cross-link), and ground-to-air transmission tests. Engineering tests of the Terragraph multi-node wireless system were conducted in San Jose, and the Open/R routing platform was open-sourced.

Nov 2017Open/R: Open routing for modern networksThis post announces the open-sourcing of Open/R, an extensible network routing platform designed for rapid innovation in network functions and applications. Open/R is being used in Facebook's backbone, data center, and wireless mesh networks, supporting various topologies and hardware. It provides a platform for disseminating network state and building new applications, with northbound interfaces for external controllers. Key features include automatic IP prefix allocation, RTT-based cost metrics, graceful restart, and fast convergence. The post highlights external adoption by partners like WiLine Networks and encourages broader community engagement.

Aug 2017FCR: Open source command runner for network devicesIntroduced and open-sourced FBNet Command Runner (FCR), a service with a Thrift interface for running commands on network devices. FCR addresses challenges in automating CLI management across multiple vendors and a large number of devices by abstracting vendor-specific details, maintaining a device database for efficient attribute retrieval, and providing scalable load balancing. It enables parallel command execution on thousands of devices, simplifying monitoring, provisioning, and deployment.

Aug 2017Steering oceans of content to the worldThis post introduces Edge Fabric, a system developed to address limitations in traditional routing protocols like BGP for managing egress traffic at Facebook's Points of Presence (PoPs). Edge Fabric augments BGP with measurement and control mechanisms to overcome its lack of congestion and performance awareness. It collects data on routes, traffic volume, and egress interface capacity. A controller (allocator) projects interface utilization and generates overrides to shift traffic away from overloaded interfaces by injecting BGP updates with high local preference. The BGP Injector service enacts these overrides. The system aims to improve network efficiency and performance by considering factors beyond BGP's static routing decisions.

May 2017Networking @Scale 2017 RecapThis post recaps the third Networking @Scale conference, highlighting technical talks on network engineering, services, IPv6, and hardware innovations. Specific contributions include details on Facebook's Express Backbone (SD-WAN long-haul network), automation and scaling of edge networks with Robotron, Alibaba's high-performance packet processing platform, Netflix's challenges in delivering terabits of content globally, Facebook's work on Zero Protocol and TLS 1.3 for low-latency transport security, CDN innovations for cache priming, certificate transparency efforts, internet-scale virtual networking using IPv6 ILA, IPv6 delivery on Amazon VPC, Microsoft's path to an IPv6-only corporate network, high-altitude platforms for rural backhaul, Equinix's automation for data center interconnection, and Facebook's custom hardware and network design.

May 2017Building Express Backbone: Facebook’s new long-haul networkThis post details the design and implementation of Facebook's Express Backbone (EBB), a new long-haul network designed to handle the rapidly increasing bandwidth demand for cross-data center replication of rich content. EBB separates inter-data center traffic from internet-facing traffic, optimizing them individually. Key technical contributions include: a hybrid traffic engineering (TE) model combining distributed control agents (Open/R) with a central controller, a four-plane physical topology inspired by data center fabrics, the use of commodity network gear by keeping the number of network prefixes small, and an iterative development approach allowing for fractional capacity deployment and A/B testing between planes. The control stack comprises BGP route injectors, an sFlow collector for traffic matrix estimation, a TE controller for path computation, Open/R agents for IGP and messaging, and LSP agents for programming forwarding tables. The controller operates in periodic cycles, computing optimal routes based on traffic demand and network state, and programming MPLS segment routes using segment routing (SR) at the headend.

Apr 2017Enhancing high-resolution 360 streaming with view predictionThis post details three technologies to enhance high-resolution 360 streaming with view prediction: a gravitational view-prediction model using physics and heatmaps, an AI model for inferring interesting video parts without heatmap data, and content-dependent streaming that improves resolution on non-VR devices. The gravitational predictor uses a physics simulator to model a marble's movement on a heatmap-derived landscape, balancing user kinetic momentum with content saliency. The AI model uses fully convolutional neural networks trained on original videos and behavioral signals to generate saliency maps. Content-dependent streaming transforms frames based on saliency maps to deliver high quality in the direction of interest within a single stream, enabling offline playback and reducing bit rates.

Mar 2017Designing 100G optical connectionsThis post details the design and deployment of 100G single-mode optical transceivers for Meta's data centers, contributing to the Open Compute Project with the CWDM4-OCP specification. It addresses the challenges of upgrading data center networks to higher speeds while maintaining low power consumption and efficiency. Key contributions include optimizing transceiver specifications for data center environments (reduced reach, link budget, and temperature range), leveraging single-mode fiber for future-proofing, and collaborating with suppliers to achieve scale on schedule.

Feb 2017End-to-end optimizations for dynamic streamingThis post details end-to-end optimizations for dynamic streaming of 360 video, building upon previous work. Key contributions include: 1. New variable offset cubemaps for encoding, which allow for smoother quality distribution by allocating more pixels to the viewport and reducing peripheral resolution, achieving up to a 92% reduction in resolution. 2. Improved reliability by considering six factors (resolution, video quality, current view orientation, stream viewport direction, length of the offset, and field of view) for stream selection, and optimizing chunk handling by estimating bandwidth on shorter intervals, measuring CDN response time, and prefetching initialization and index chunks. 3. Player performance boosts through filtering stream candidates early, pre-computing intensive computations, and using lookup tables. These optimizations collectively led to up to a 92% decrease in bit rates and an 80% decrease in interruptions.

Feb 2017Spatial audio — bringing realistic sound to 360 videoIntroduced new 8-channel hybrid higher-order ambisonics encoding and rendering technology for 360 video, optimizing for bandwidth and quality. Developed a spatial audio renderer with sub-millisecond latency, supporting both spatialized and head-locked audio simultaneously. Created the Spatial Workstation tool suite for creators and the FB360 Encoder for seamless upload to Facebook. The audio engine is C++ based with optimized vector instructions, compiled to asm.js for web, maintaining a unified codebase and low binary size (~100KB). Addressed MP4 container limitations by using three audio tracks and XML metadata within an MP4 box for forward extensibility.

Jan 2017Disaggregate: Networking recapThis post is a recap of the 'Disaggregate: Networking' conference, detailing presentations on open and disaggregated networking solutions. Key contributions include discussions on Facebook's custom networking hardware (Wedge 100, Backpack, Voyager) and software (FBOSS, OpenBMC), the Robotron system for hybrid network management, programmable forwarding planes (Tofino, P4), and the broader open networking ecosystem with contributions from various vendors. It highlights the benefits of decoupling hardware and software for increased efficiency, flexibility, and scalability in data center networks.

Jan 2017Building Zero protocol for fast, secure mobile connectionsThis post details the development and deployment of Zero protocol, an experimental 0-RTT security protocol over TCP based on QUIC's crypto protocol. It addresses the high latency of TLS 1.2 connection establishment, particularly in emerging markets. Key technical contributions include modifications to QUIC key derivation to fix a reuse vulnerability, in-band server config rotation for efficiency, retry safety optimizations, time-bound 0-RTT data with clock skew correction, and modifications for TCP compatibility by removing explicit sequence numbers and adding an explicit length field. The post also discusses deployment considerations, including port selection and the design of the Zero RTT API within Mobile Proxygen.

Jan 2017This browser tweak saved 60% of requests to FacebookThis post details Meta's collaboration with Chrome and Firefox to improve browser caching efficiency. It explains the concepts of revalidation, content-addressed URLs, and the problem of unnecessary conditional requests during reloads. The post highlights specific contributions: working with Chrome to fix an issue where POST requests triggered unnecessary revalidation and to change reload behavior to avoid revalidating sub-resources with long max-age, and proposing the `cache-control: immutable` header to Firefox, which was adopted to prevent revalidation of resources guaranteed not to change. These efforts resulted in a 60% reduction in static resource requests and improved page load times.

Jan 2017The growing ecosystem around open networking hardwareThis post details the growing ecosystem around Meta's open networking hardware initiatives, including Wedge 100, Backpack, and Voyager. It highlights contributions to the Open Compute Project and Telecom Infra Project, and showcases how various companies (Cumulus Networks, Big Switch Networks, Barefoot Networks, SnapRoute, Canonical, Cavium, Apstra) are building software and hardware solutions that leverage these open designs. The post emphasizes the benefits of disaggregation and openness in enabling faster evolution, flexibility, and scalability of data center infrastructure. It also announces the commercial availability of Wedge 100 and pre-orders for Backpack.

Jan 2017Legacy support on IPv6-only infraThis post details the technical implementation of supporting IPv4 external requests on an IPv6-only data center infrastructure. It describes modifications to Layer 4 (L4LB/shiv) and Layer 7 (L7LB/proxygen) load balancers, including the use of IPv4 link-local addresses as BGP next-hops for IPv6-only servers, tunneling IPv4 traffic inside IPv6 using a custom kernel module for decapsulation, and the use of gnlpy to configure IPVS forwarding. It also explains how Direct Server Return (DSR) is maintained for IPv4 responses.

Dec 2016Favorite hacks of 2016This post details several hackathon projects that contributed to Meta's engineering capabilities. The 'Voyager' transponder prototype demonstrated 200GB ethernet traffic transmission over a single wavelength, building on data center technology and later contributed to the Telecom Infra Project. Another hack leveraged Facebook Live infrastructure for private live-streaming within Messenger, exposing the multiway service via GraphAPI for third-party WebRTC clients, which influenced Oculus Parties. Safety Check saw improvements with custom notifications and a Community Help module. Account Kit introduced instant verification using existing Facebook sessions to bypass SMS confirmation. Nuclide gained Swift support, and an 'Atom in orbit' hack aimed to run Atom in the browser for remote development. Finally, 'Adaptive attachments' extended the Recommendations engine to support photo uploads to shared albums, crowdsourced videos, and collaborative games via post comments.

Nov 2016Facebook demonstrates record-breaking data rate using millimeter-wave technologyThis post details the demonstration of a record data rate of nearly 20 Gbps over 13 km using millimeter-wave (MMW) technology. It highlights the use of custom-built, high-linearity RF components and custom-built MMW power amplifiers with high-efficiency power combining techniques. The post also describes the development of post-amplification spectrum multiplexing to improve power amplifier efficiency and the use of conical scanning for high-precision antenna pointing and calibration using sun tracking. The technology is intended for terrestrial backhaul, backup for optical solutions, and as a link between ground stations and the Aquila UAV.

Nov 2016Introducing Backpack: Our second-generation modular open switchThis post introduces Backpack, Meta's second-generation modular open switch platform, designed to support 100G data center requirements. It features a fully disaggregated architecture with switch elements, separation of data, control, and management planes, and an orthogonal direct chassis for improved signal integrity and thermal performance. Backpack utilizes FBOSS and OpenBMC for its networking stack and system management, and integrates BGP for routing with fabric cards acting as route reflectors. The post details the rigorous three-phase testing approach (engineering validation, design validation, and production validation) and highlights improvements made over the first-generation 6-pack switch, such as removing BGP Next Hop from ECMP groups and implementing port striping for better load balancing. It also emphasizes the importance of control and data plane monitoring using tools like Scuba and a new FIB monitor.

Nov 2016An open approach for switching, routing, and transportThis post introduces Open Packet DWDM, a new approach combining packet and DWDM technology for metro and long-haul fiber optic transport networks. It details the Voyager transponder, the industry's first 'white box' optical packet transport solution, which separates software and hardware based on open specifications. The post highlights the use of Broadcom Tomahawk switch ASIC and Acacia Communications' DSP ASIC, and discusses the software architecture for configuration, management, and monitoring. Field trials with Equinix and MTN are mentioned, and the plan to open-source Voyager software and contribute to TIP's Backhaul: Open Optical Packet Transport (OOPT) project group is announced.

Oct 2016Networking @Scale BostonThis post details presentations from the Networking @Scale Boston conference, highlighting advancements in network routing platforms (Open/R), cloud networking solutions (Akamai's SureRoute IP, Optimized Transport), browser networking performance, modeling operational complexity, scaling Facebook Live traffic with techniques like request coalescing and predictive splines, network traffic visibility and anomaly detection, scaling traffic configurations, securing microservices with encryption (Kerberos, TLS), gigabit wireless networks (Terragraph), and the deployment of QUIC for web performance and video streaming.

Oct 2016Client-side ranking to more efficiently show people stories in feedIntroduced a new client-side ranking architecture for News Feed. This architecture allows stories to be re-ranked on the client after being sent from the server, ensuring that stories with fully-loaded media are prioritized and that users see relevant content more quickly, even on slow network connections. This avoids spinners and gray boxes by requiring stories to have all necessary media available before rendering and by optimizing content for each session.

Oct 2016Wedge 100: More open and versatile than everThis post details the development and open-sourcing of Wedge 100, a second-generation 100G top-of-rack network switch, and its integration into Meta's data center strategy. It highlights hardware updates for improved serviceability and thermal management, and the reuse of the FBOSS software stack with adaptations for 100G ASIC chips. The post also discusses the operational challenges of managing parallel 40G and 100G networks and the flexibility gained from owning the entire software stack, including OpenBMC for board management. The availability of Wedge 100 as a commercial product and the growing ecosystem of software solutions built on top of it are also emphasized.

Sep 2016DHCPLB: An open source load balancerThis post introduces DHCPLB, an open-source load balancer designed to address traffic imbalance issues in Facebook's DHCP infrastructure. It details the problem of uneven DHCP request distribution caused by network topology and ECMP limitations, leading to incidents where servers were flooded. DHCPLB was developed to sit behind anycast VIPs and distribute DHCPv4 and DHCPv6 requests among KEA servers. Key features include DHCPv6 support (RFC3315), A/B testing capabilities for KEA server versions, MAC address-based overrides for debugging, integration with Scuba for monitoring, hot-reloading of configuration files, BGP advertisement via ExaBGP, and two balancing algorithms (Round Robin and ModuloN based on Transaction ID). The post also outlines the production rollout and architecture, including TOR relay configuration and DHCPLB deployment.

Sep 2016Facebook announces new tech at @Scale 2016This post announces the development and availability of Zstandard, a new compression algorithm that offers smaller data sizes and faster compression/decompression speeds compared to industry norms. It also introduces MyRocks, an open-source MySQL storage engine built on RocksDB, which demonstrates significant storage reduction (50%) on flash storage and improved efficiency. The post also details a new 360 video stabilization technology that combines 2D and 3D stabilization with a 'deformed rotation' motion model to improve video quality and reduce bit rates. BrowserLab, an automated testing environment for web regressions, is presented as a tool to detect performance issues as small as 20ms. Finally, Infer, Facebook's open-source static analyzer for Java, C, and Objective-C, is highlighted for its extensibility and ongoing support for C++.

Aug 2016BrowserLab: Automated regression detection for the webThis post introduces BrowserLab, an automated system for detecting client-side performance regressions in web applications. It details the challenges of measuring client performance at scale, including sources of noise like environment, data dynamism, and code non-determinism. The system design incorporates an HTTP proxy to isolate server noise, RAM disks to reduce client environment noise, a backend proxy for consistent data views, and deterministic replacements for random functions in client and server code. The experiment procedure involves recording and replaying requests to achieve high accuracy (20ms with 95% confidence). BrowserLab is integrated into CI pipelines for automated regression detection and bisecting to identify problematic commits.

Aug 2016360 video stabilization: A new algorithm for smoother 360 video viewingIntroduced a new algorithm for 360 video stabilization using a hybrid 3D-2D approach and a novel 'deformed-rotation' motion model. This algorithm improves efficiency by reducing bit rate (10-20%) and enables faster stabilization (under 22ms per frame). Developed a 360-hyperlapse algorithm as an extension for accelerated playback. The technical details include converting equirectangular projection to cube maps for feature point tracking, using KLT tracker and OpenCV, estimating relative rotations between key frames with Nister's five-point algorithm in RANSAC via OpenGV, and optimizing in-between frames with a Ceres-solver-based non-linear robust least-squares problem using a deformable-rotation model. Performance optimizations include computing warp field coordinates for every 8x8 pixels and interpolating bilinearly for CPU-only execution.

Aug 2016Robotron: Top-down network management at scaleThis post introduces Robotron, a top-down network management system designed to manage tens of thousands of network devices and hundreds of thousands of servers globally at Facebook. Robotron aims to reduce manual interaction with network devices for tasks like circuit turn-up, migration, device provisioning, OS upgrades, ACL modification, protocol tuning, and monitoring. It evolved from an earlier system, FBNet, which started as an object store for basic network device information and evolved to store normalized, vendor-agnostic views of network state. Robotron covers network design, configuration generation (vendor-specific), deployment (phased rollouts), and monitoring. The system translates high-level engineer intent into FBNet objects, then generates vendor-specific configurations, deploys them safely, and monitors for deviations from the desired state. The post highlights the challenges of distributed configurations, multiple domains, versioning, dependencies, and vendor differences, and mentions the open-sourcing of parts of Robotron like fbpush.

Jun 2016Developing the technical language behind creativity for CanvasThis post details the technical design and implementation of Canvas, a platform for creating full-screen, immersive mobile content experiences. It describes the UI challenge of condensing creative requests into a set of high-level primitives (autoplay videos, photos, tilt-to-pan media, carousels, buttons, headers, rich text) and the API/language designed for configuring these components and defining their layout. Key technical aspects include leveraging existing infrastructure for video autoplay and rendering/layout engines (like AsyncDisplayKit), a declarative GraphQL-like language for defining component composition and attributes, a style system analogous to CSS classes (e.g., FIT_TO_HEIGHT), and an action system for interactive elements (Open URL, Post Data, Activate Component, Reload Canvas). The post also highlights the technical challenges and solutions related to mobile-first design, optimizing for loading speed and time to interact, and the real-time mobile preview mechanism using deep links.

Jun 2016Growing the Wedge/Wedge 100 communityThis post details the growth of the Wedge and Wedge 100 community, highlighting the contributions of various companies and startups in developing software for these open, disaggregated network switches. It showcases how AT&T, SK Telecom, and Canonical have leveraged Wedge for higher-layer networking services, OpenFlow control, and server management respectively. The announcement of SnapRoute's FlexSwitch software on Wedge and Wedge 100 signifies a significant step in providing a full L2/L3 open-source networking stack for these platforms, further fostering an open and disaggregated networking ecosystem.

May 2016Networking @Scale, May 2016 — RecapThis post recaps the second annual Networking @Scale conference, highlighting technical discussions and presentations from various companies including Microsoft, Google, AT&T, Netflix, JPL, and Facebook. Key topics covered include data center network monitoring (PingMesh, NetBouncer), wide area networks (Google's B4), wireless connectivity for urban environments (Facebook's Terragraph), scaling live video streaming (Facebook Live), interplanetary networking (JPL), modular routing platforms (Facebook's Open/R), EVPN for large-scale service overlays (AT&T), and application data caching (Netflix's EVCache). Day two focused on IPv6 adoption and implementation, with presentations on IPv6 buildouts, address space visualization, and backbone/data center conversion to IPv6.

May 2016Introducing Open/R — a new modular routing platformThis post introduces Open/R, a new modular and extensible distributed network application platform. It was initially designed as a shortest-path routing system for Terragraph, a multi-node wireless network for urban internet connectivity. Open/R's modular design allows for additional applications beyond routing, and it has been adapted for other networking infrastructure. The platform generalizes replicated state databases and uses Thrift for message encoding and ZeroMQ for message exchange. Key modules include KV-STORE for publishing and subscribing to key-value pairs, LINK MONITOR for detecting link status changes, DECISION for computing shortest paths using Dijkstra's SPF algorithm and loop-free alternatives, and FIB for interacting with the hardware forwarding information base. Open/R has been scaled to networks with thousands of nodes and validated through extensive emulation test-beds. The company plans to contribute it to the open source community and TIP.

Apr 2016Wangle — an asynchronous C++ networking and RPC libraryThis post introduces Wangle, an asynchronous C++ networking and RPC library inspired by Netty and Finagle. It details Wangle's asynchronous building blocks (folly/io/async, Futures), concurrency primitives (CPUThreadPoolExecutor, IOThreadPoolExecutor with LIFO semaphores and MPMC queues), and the pipeline/codec concept for data transformation. It also covers ServerBootstrap and ClientBootstrap for managing connections and provides a concrete example of a file streaming server that offloads blocking disk reads to the global CPU pool.

Mar 2016Opening designs for 6-pack and Wedge 100This post details the opening of designs for the 6-pack modular switch and the Wedge 100 TOR switch, a critical step towards supporting 100G connectivity in Meta's data centers. It describes the hardware specifications of Wedge 100 (Broadcom Tomahawk ASIC, OpenRack v2, COM-E, QSFP28 DAC/CWDM4 optics) and its extensive testing for various connectivity modes. It also elaborates on the 6-pack's architecture, its role as a fabric switch (FSW) in the data center fabric, and the significant increase in scalability and performance requirements compared to top-of-rack switches. The post details the rigorous network-level testing performed on 6-pack, including functionality validation (BGP peerings, route policy, ECMP) and failure condition testing (link failures, card insertions/removals, process restarts), demonstrating sub-2ms failure recovery. It also discusses the software architecture of FBOSS on 6-pack, highlighting the decision to avoid a central supervisor module and instead treat the chassis as a collection of connected Wedges.

Feb 2016Performance @Scale 2016 recapThis post summarizes the "Performance @Scale 2016" event, highlighting discussions on low-level system profiling, production measurements, regression detection, and efficient triage. Key technical takeaways include: Brendan Gregg's presentation on using BPF for Linux performance analysis, Ben Maurer's overview of Facebook's web performance challenges and solutions (Time-to-Interact, Display Done metrics, Early Flush, Big Pipe, Bootloader), Guilin Chen's description of Facebook's AutoTriage system for automatic regression detection, Daniel Greenia's insights on using performance analysis to improve ad revenue by detecting fraud, Vance Morrison's approach to client-side performance analysis with PerfView, Jim Roskind's discussion on high-performance networking in Chromium and the evolution of QUIC, Michelle Filiba's introduction to mobile performance engineering with the Loom telemetry framework, and Anant Rao's visualization techniques for real user performance on mobile using RUMv2 and anomaly detection.

Feb 2016Gear VR to get Dynamic Streaming for 360 videoThis post announces the upcoming integration of Meta's dynamic streaming technology for 360 video into Gear VR. This technology aims to deliver 360 streaming video more efficiently, resulting in clearer visuals and faster playback for VR experiences.

Feb 2016NetNORAD: Troubleshooting networks via end-to-end probingIntroduced NetNORAD, an end-to-end probing system for network troubleshooting that treats the network as a black box, independent of device polling. It measures packet loss ratio and latency by having servers ping each other using UDP probes. The system categorizes network issues by proximity (DC, Region, Global) and uses time-series analysis for alarm generation. It incorporates outlier detection for fault isolation and complements fbtracert for hop-by-hop analysis. Key components (UDP pinger/responder and fbtracert) were open-sourced.

Dec 2015Continuing to build News Feed for all types of connectionsThis post introduces improvements to News Feed for users with slow or intermittent internet connections. The key technical contributions are: 1. Prioritizing and ranking already downloaded, relevant stories when the app starts, when navigating to News Feed, or when pulling to refresh, instead of waiting for new stories to load. This involves factoring in image availability. 2. Periodically retrieving new stories when a good connection is available to keep the feed updated. 3. Enabling users to compose and post comments while offline, with the comments being sent when a connection is re-established. These changes focus on efficient content delivery and offline functionality to improve user experience on constrained networks.

Dec 2015Under the hood: Broadcasting live video to millionsThis post details the engineering challenges and solutions for scaling live video broadcasting to millions of users. It introduces the 'thundering herd' problem and explains how edge caches, origin caches, and request coalescing are used to manage traffic spikes. It also describes the implementation of RTMP for reducing latency to a few seconds, contrasting it with HLS and highlighting the benefits of its push model and small chunk sizes for interactive live streams.

Nov 2015Optimizing Facebook for iOS start timeThis post details optimizations made to the Facebook for iOS app's start time, focusing on cold starts. Key contributions include: defining stable metrics for app start (from icon tap to feed loaded), breaking down cold start into request time, network time, and response processing time, identifying initialization as a bottleneck, refactoring feed request dependencies to allow concurrent initialization, implementing UDP priming to pre-fetch data and reduce TCP connection overhead, optimizing story fetching from a batch to a streaming approach (1+1+X), and deferring non-critical background services until after the feed is displayed to reduce contention.

Nov 2015Open networking advances with Wedge and FBOSSThis post details the deployment and operationalization of Wedge, an open top-of-rack (TOR) network switch, and FBOSS, its Linux-based control software. It highlights the open hardware design contributions to the Open Compute Project (OCP), enabling community customization. Key technical advancements include detailed monitoring for software updates (fbossdeploy), non-stop forwarding via Warm Boot, and CPU protection mechanisms to prevent control plane overload during traffic surges. Lessons learned include integrating Wedge into common service infrastructure rollouts and avoiding circular dependencies. The post also introduces the concept of "backup groups" for rack switches to handle link failures and discusses protecting the CPU from transient routing loops.

Nov 2015Milestones and plans for the OCP Networking ProjectThis post details the evolution of the OCP Networking Project, highlighting its progress in networking hardware, software, and testing. It introduces FBOSS (Linux-based network applications for switches) and OpenBMC (low-level system management software) as key software contributions. It also mentions the development of open hardware modular switches like 6-pack and the Wedge specification, emphasizing their role in data center fabrics and the broader goal of increasing network innovation through disaggregation and open source solutions.

Oct 2015Building for emerging markets: The story behind 2G TuesdaysIntroduced "2G Tuesdays" as an internal initiative to allow employees to simulate a 2G connection, fostering empathy and understanding of user experience on low-bandwidth networks. This initiative aims to inform product development decisions for emerging markets by highlighting issues and pain points experienced by users on 2G connections. This complements existing tools like Connection Class and Augmented Traffic Control.

Oct 2015Improving the Linux kernel with upstream contributionsThis post details Meta's upstream-first philosophy for Linux kernel development, contributing changes to the mainline kernel before internal deployment. Specific contributions include: improvements to blk-mq for block layer queuing, stabilization of Btrfs with over 50 changes, implementation of the New Vegas congestion control algorithm, writeback support for cgroups to enable IO limiting, advancements in network virtualization (GUE, IPv6 flow label, ILA), IPv6 performance improvements to scale the routing tree, a caching layer for MD RAID 5/6 to address the write hole and improve performance, and progress in release engineering for faster kernel deployment.

Sep 2015IPv6: It’s time to get on boardThis post advocates for the adoption of IPv6, highlighting its technical benefits over IPv4, including easier network management through auto-configuration, improved end-to-end connectivity integrity by eliminating the need for NAT devices, and enhanced interoperability and mobility. It shares Facebook's experience of achieving 10-15% faster access over IPv6 and encourages other developers to migrate to realize similar performance advantages and unlock new technological possibilities enabled by the vast IPv6 address space.

Aug 2015The technology behind preview photosThis post details the technical approach to generating and delivering low-resolution preview images (approximately 200 bytes) for cover photos. The solution involves a custom JPEG encoding scheme that omits the standard header, allowing the preview data to be embedded within the initial GraphQL response. This reduces the number of network requests and significantly speeds up the initial display of profiles and pages, especially on low-connectivity networks. The approach leverages Gaussian blur on the client-side to create a frosted-glass effect from a low-resolution source image (around 42x42 pixels) and uses a fixed Huffman table and quantization table (Q20) for efficient compression.

Jul 2015Linux IPv6 improvement: Routing cache on demandThis post details the implementation and upstream contribution of an on-demand routing cache creation mechanism for Linux IPv6. This solution addresses scalability issues in large IPv6 deployments by significantly reducing the size of the IPv6 routing tree. Key technical contributions include sharing gateway routes for peers with default PMTU, implementing per-CPU entries for gateway routes to mitigate cache bouncing, and ensuring compatibility with subsystems that expect '/128' routes. The post also highlights bug fixes related to '/128' routing entries and optimizations for `fib6_lookup()`.

Jul 2015Using ISC Kea DHCP in our data centersThis post details Facebook's migration of its data center DHCP infrastructure from ISC dhcpd to ISC Kea. The migration was driven by the need for faster hardware bootstrapping and reduced reconfiguration times, which were problematic at scale with the old system. The team leveraged Kea's hook points to customize its behavior and developed a stateless DHCP service that dynamically fetches configuration from an inventory system, simplifying deployment and maintenance.

Apr 2015Web performance: Cache efficiency exerciseThis post details a study conducted by Meta to measure the cache efficiency of facebook.com, comparing current performance to an older Yahoo study from 2007. It describes the methodology for recreating the study using a dedicated PHP endpoint and an image tag next to the Facebook search bar. The results show a cache hit rate of 84.1% (excluding Firefox v32+), an improvement over the 2007 study, but also highlight that caches do not stay populated for long, with a 42% chance of a desktop request having a cache that is at most 47 hours old. The post discusses practical applications and best practices for utilizing browser caches, concluding that despite frequent code releases, cache performance has improved and remains a critical factor for web performance.

Mar 2015Classes: Performance and network segmentation on AndroidIntroduced and open-sourced two tools for Android: Year Class, which classifies devices based on hardware specifications to a year, and Connection Class, which segments real-time network performance into four categories (Poor, Moderate, Good, Excellent) based on bandwidth and latency measurements. These tools are designed to help developers analyze and adjust feature performance at runtime.

Mar 2015Augmented Traffic Control: A tool to simulate network conditionsThis post introduces Augmented Traffic Control (ATC), a tool developed to simulate various network conditions (2G, Edge, 3G, LTE) for testing and optimizing Facebook's applications. The development involved an initial attempt to create a 2G network using base stations and mobile phones, which proved problematic due to hardware acquisition challenges. The team pivoted to building a Wi-Fi network that could be configured with different simulation speeds. ATC was integrated into the Facebook campus wireless network and its APIs are used in automation. The post details how ATC helped teams like Messenger and download functionality teams identify weaknesses, save development time, and reduce the need for extensive A/B testing. The design for ATC has been open-sourced.

Mar 2015Facebook Open Switching System (“FBOSS”) and Wedge in the openThis post announces the initial release of the Facebook Open Switching System (FBOSS) project on GitHub and the proposed contribution of the Wedge top-of-rack switch specification to the OCP networking project. It details the anatomy of a network switch, explaining the role of ASICs and the FBOSS agent in programming them. FBOSS is presented as a set of applications on Linux, not a full OS, categorized into low-level, automation, and control apps. The FBOSS agent's lean feature set, focused on L3 unicast routing with ECMP and BGP, is highlighted. The post also discusses the challenges of open-source switching software due to proprietary ASIC SDKs and the role of Broadcom's OpenNSL APIs in enabling the open-sourcing of the FBOSS agent.

Feb 2015Introducing “6-pack”: the first open hardware modular switchIntroduced "6-pack", the first open hardware modular switch platform, designed to address the limitations of traditional networking technologies at scale. "6-pack" is a full mesh non-blocking two-stage switch with 12 independent switching elements, each capable of 1.28Tbps switching capacity. It leverages the "Wedge" switch as a building block and integrates a Linux-based OS (FBOSS). The platform uses a hybrid SDN approach with a split control plane for management and stability. It is designed for modularity, allowing for customization of radix and other configurations using common building blocks. The fabric card aggregates management networks and enables local full mesh creation. "6-pack" is in production testing and planned for contribution to the Open Compute Project.

Jan 2015Faster Photos in Facebook for iOSThis post details the adoption of Progressive JPEG (PJPEG) in Facebook for iOS to improve photo loading times and reduce data consumption. Previously, the app downloaded multiple versions of an image (small and full-size) and displayed the small one while waiting for the larger one. PJPEG allows the app to render an image progressively as scans of increasing detail are downloaded, reducing the need to download multiple versions, thus saving data (approx. 10%), reducing network connections, decreasing disk storage, and improving perceived wait time by 15%. The trade-off of increased CPU usage for decoding was managed by balancing data usage, network latency, and CPU utilization.

Nov 2014Introducing data center fabric, the next-generation Facebook data center networkThis post introduces the 'data center fabric' as a next-generation network architecture for Meta's data centers, moving away from traditional cluster-based designs. It details a disaggregated approach using 'pods' as uniform network units, connected by spine switches across independent planes. The fabric utilizes standard BGP4 for routing with centralized override, is all layer3 and dual-stack (IPv4/IPv6), and heavily employs ECMP with flow-based hashing. It allows for gradual scalability from 4:1 to 1:1 fabric oversubscription by adding spine switches. The physical infrastructure is optimized for rapid deployment with shorter cabling lengths.

Nov 2014Solving the Mystery of Link Imbalance: A Metastable Failure State at ScaleThis post details the discovery and resolution of a metastable link imbalance failure state in Facebook's aggregated network links. The root cause was identified as an interaction between a Most Recently Used (MRU) connection pool strategy in the TAO graph store and transient network congestion. The MRU pool, when faced with bursts of database queries, would preferentially select connections that had recently experienced latency (due to congestion), thereby exacerbating and sustaining the imbalance. The fix involved switching to a Least Recently Used (LRU) connection pool with a maximum connection age, which significantly improved traffic distribution and eliminated the metastable failure.

Nov 2014Year class: A classification system for AndroidThis post introduces the 'year class' classification system for Android devices, which categorizes phones based on their hardware specifications (RAM, CPU cores, clock speed) to represent their historical high-end status. This system allows for more accurate performance analysis and targeted feature rollouts, such as shipping a better commenting UI to higher-class phones while optimizing for older models. It also influences News Feed ranking to avoid showing video stories on devices that cannot play them smoothly. The post also details the development of a 'connection quality' score, which measures network latency and image transfer rates to dynamically adjust app features like video autoplay and photo quality, independent of reported network types, to ensure graceful performance on various network conditions, including congested EDGE connections.

Nov 2014Introducing Proxygen, Facebook’s C++ HTTP frameworkThis post announces the open-sourcing of Proxygen, a C++ HTTP framework developed by Facebook. Proxygen supports HTTP/1.1, SPDY/3, SPDY/3.1, and is developing HTTP/2 support. It is designed as a high-performance, easy-to-integrate framework with sensible defaults, differentiating itself from highly configurable servers like Apache and Nginx. The motivation for building Proxygen included deep integration with Facebook infrastructure (e.g., Thrift, ODS), code reuse for event-driven networking libraries, addressing scale limitations of existing stacks, and implementing specific features like SPDY, WebSockets, and custom load-scheduling algorithms. The architecture is described as a four-part abstraction: session, codec, transaction, and handler, enabling support for multiplexing protocols. It utilizes C++ features, Thrift, and Folly, and employs non-blocking I/O with epoll for efficiency. An echo server example demonstrates its usability and performance, with benchmarks showing efficiency for binary protocols. The impact highlights Proxygen's role in rapid feature iteration and experimentation, such as deploying HPACK on SPDY. The post also mentions the open-sourcing of Thrift alongside Proxygen.

Oct 2014Building Mobile-First Infrastructure for MessengerThis post details the development of a mobile-first infrastructure for Messenger, introducing a new sync protocol that reduced non-media data usage by 40% and errors by 20%. Key technical contributions include: a shift from a pull-based to a push-based snapshot + delta model using MQTT for real-time updates, replacement of JSON with Thrift for a 50% reduction in payload size, and the creation of the Iris service. Iris acts as a totally ordered queue for messaging updates, enabling tiered storage with recent messages in memory, a week's worth in a MySQL/flash backing store, and older history on traditional disk storage. This architecture decouples real-time message delivery from disk write latencies, improving reliability and speed.

Sep 2014Introducing mcrouter: A memcached protocol router for scaling memcached deploymentsThis post introduces mcrouter, a memcached protocol router designed to scale memcached deployments. It details mcrouter's features such as standard protocol support, connection pooling, multiple hashing schemes (including consistent hashing), prefix routing, replicated pools for availability and read rate handling, production traffic shadowing for testing new hardware, online reconfiguration without client latency, flexible routing via composable route handles (e.g., all-sync, all-majority, failover), destination health monitoring with automatic failover (distinguishing soft and hard errors), cold cache warmup, broadcast operations, reliable delete streams, multi-cluster support, rich stats and debug commands, quality of service features like request throttling and rate limiting, large value handling, multi-level caches, IPv6 and SSL support, and a multi-threaded architecture. The implementation uses C++11, Folly, fbthrift, libevent, and a custom fiber library. It also mentions its use by Instagram and a beta test with Reddit.

Jun 2014Improving Facebook on AndroidThis post details significant improvements made to the Facebook Android app focusing on performance, data efficiency, networking, and application size. Key contributions include: deferring non-critical initializations to improve startup times, fetching News Feed stories earlier for faster loading on poor networks, optimizing image downloads using WebP compression (25-35% savings over JPG) and adaptive resolution loading based on viewport size, and fine-tuning image caching strategies. The networking stack was enhanced by adopting OkHttp, enabling faster retries and SPDY protocol support, leading to a 90% reduction in slow/failed image loads. Application size was reduced by 65% by utilizing Google Play's ability to upload multiple APKs for different device configurations. Automated tests and instrumentation were developed to monitor and prevent regressions in data usage and app size.

Jun 2014Introducing “Wedge” and “FBOSS,” the next steps toward a disaggregated networkIntroduced "Wedge," a new disaggregated top-of-rack network switch hardware design, and "FBOSS," a new Linux-based operating system for network switches. Wedge leverages a modular microserver architecture similar to existing server designs, allowing for flexibility in processor choice (Intel, AMD, ARM) and integration into distributed fleet management systems. FBOSS aims to accelerate development by leveraging existing server management software, providing a Thrift-based abstraction layer for switch ASIC APIs, and enabling a hybrid control plane for optimized network path selection and resource utilization. This initiative aims to make network operations more akin to server operations, improving visibility, automation, and control.

Mar 2014Looking back on “Look Back” videosThis post details the engineering challenges and solutions for scaling the 'Look Back' video feature, focusing on the infrastructure required to generate, store, and deliver over 720 million videos. Key contributions include detailed planning for storage (11 petabytes), computing capacity, and network bandwidth (450 Gbps peak outgoing). It highlights the need for rapid hardware provisioning, data center power management, and CDN partner coordination to handle unprecedented traffic spikes. The post also touches upon the organizational aspect of mobilizing dozens of teams to achieve this rapid deployment.

Feb 2014Under the Hood: Building and open-sourcing fbthriftThis post details the evolution of Thrift into fbthrift, an internal, performance-optimized branch. Key contributions include the introduction of a new C++ code generator (cpp2), the THeader protocol for per-request features (like compression and tracing), and the adoption of folly's IOBuf for improved asynchronous workload performance and memory management. These changes have resulted in significant latency reductions (up to 50%) and decreased memory footprints for services using fbthrift.

Nov 2013The Mature Optimization HandbookThis post introduces 'The Mature Optimization Handbook,' which provides a framework for performance measurement and optimization. It emphasizes the importance of data-driven decisions, avoiding premature optimization, and focusing on the critical 3% of code where optimization yields the most significant gains. It also highlights that performance optimization is a cost/benefit decision, best applied to mature systems, and can be a crucial feature when resources are limited or post-ship changes are difficult.

Jun 2013Adding :face: to every IP: Celebrating IPv6’s one-year anniversaryThis post details Meta's internal adoption of IPv6, marking a significant step in preparing their network infrastructure for future growth and flexibility. It highlights the transition from limited IPv6 user adoption (under 1% to over 4%) to internal network-wide support. Key technical challenges addressed include ensuring software compatibility with both IPv4 and IPv6, managing the complexities of IPv6 address representation and storage, and overcoming bugs and scaling issues due to less extensive large-scale IPv6 usage and testing. The network engineering team's efforts to achieve full IPv6 support down to every host are emphasized, along with industry collaboration to make IPv6 ubiquitous.

May 2013Speeding up the Activity social pluginThis post details the optimization of the Activity social plugin, resulting in a 5x reduction in payload size (from 257KB to 48KB) and a decrease in HTTP requests from 18 to 5. Load times improved by nearly 2x (0.6s vs. 1.14s), with initial painting time nearly halved (0.38s vs. 0.68s). Key technical contributions include rewriting and reducing JavaScript and CSS sizes, untangling dependencies, improving packaging, and inlining all CSS for faster initial rendering.

Apr 2013Under the Hood: The natural language interface of Graph SearchThis post details the development of a natural language interface for Graph Search, which leverages a structured graph database. The system employs entity recognition, lexical analysis, and semantic parsing with a weighted context-free grammar to understand user intent. It handles variations in user input, including grammatical errors and synonyms, and uses a cost structure for ranking interpretations. The underlying infrastructure for Graph Search, including indexing and ranking, has been previously described.

Feb 2013Like Box plugin: Now 2-4x fasterThe Like Box plugin was rewritten from the ground up to improve performance. Key changes include: new, smaller HTML and CSS components; inlining small CSS for faster rendering; reduced overall JavaScript size; better JavaScript packaging into a single file loaded via a single HTTP request; and asynchronous, non-blocking JavaScript loading. These optimizations resulted in a 2-4x speed improvement, a reduction from 15 resources to 4, and a 5x reduction in payload size for the simplest configuration. Perceived performance improvements were even more significant, with first paint times improving by up to 7x.

Sep 2012A faster, better link shimIntroduced a new link shim implementation that pre-checks all links on a page for maliciousness before they are sent to the browser, rather than checking only clicked links. This change eliminates the extra round trip to Facebook's servers for each click, saving approximately one second per external link click. Leveraged the 'meta referrer' tag (available in Chrome 17+ and Safari 6+) to restrict the amount of source URL information shared with external sites via the Referer header, balancing the need to indicate traffic origin with privacy concerns. For browsers not supporting 'meta referrer', the older, slower system is used.

Jun 2012Under the Hood: Network implementation for World IPv6 LaunchThis post details the implementation of Layer 3 DSR for IPv4 and IP6-in-IP6 tunneling for IPv6 to support World IPv6 Launch. It describes the challenges of moving from Layer 2 DSR to Layer 3 DSR, the use of IP-in-IP tunneling to achieve direct server return across L3 domains, and the specific kernel patch required to enable IP6-in-IP6 decapsulation on the fallback device. The post highlights the successful integration of IPv6 as a first-class citizen in their infrastructure, enabling connections from various endpoints.

Mar 2012Under the Hood: Building the Location APIIntroduced a universal location tagging system for all content types, moving beyond mobile-specific check-ins. Developed a global places directory with a new indexing strategy (multi-polygon and prefix-based) for efficient global search and relevance ranking. Designed and implemented the timeline map feature to visualize location-tagged content. Created aggregation algorithms for map pins and systems for suggesting locations for existing albums. Launched location APIs for third-party applications to add and read location data.

Dec 2011Meet a Facebook Engineer: Carlos BuenoThis post introduces Doppler, a system for measuring worldwide packet latency using DNS hacks and JavaScript. It maps user IPs to DNS resolver IPs, providing a novel approach to network performance analysis that works across various devices, including smartphones. The engineer also discusses the ongoing effort to achieve "0 degrees Kelvin" in profiling tools for precise system measurement and the importance of basic statistics and experiment design for engineers.

Jul 2011Doppler: Internet RadarThis post introduces Doppler, an internal project for mapping and measuring the network between users and Facebook. It details the methodology of using a lightweight JavaScript probe appended to page hits to initiate HTTP requests to tiny images on servers, timing their load times. Specific experiments are described for measuring end-to-end DNS latency (by using unique hostnames) and packet Round Trip Time (RTT) by subtracting the time for a cached request from a full request. The collected data, billions of measurements over a year, is used to generate country-level and Autonomous System Number (ASN)-level graphs to identify network issues. Doppler is also used for evaluating TCP optimizations and new network hardware, and is being extended to support mobile users, SSL, and IPv6. Future work includes using Doppler data for DNS-based global server load balancing by indexing latency by resolver IP.

Jun 2011Exciting Results from World IPv6 DayThis post details Meta's successful participation in World IPv6 Day, where they enabled IPv6 on their site for 24 hours. They observed over 1 million users connecting via IPv6 with no significant increase in Help Center requests, indicating successful infrastructure readiness. Based on these results, they decided to leave their Developer site dual-stacked and continue adapting their codebase to support IPv6, highlighting a commitment to advancing the protocol for internet growth.

Jun 2011Facebook and World IPv6 DayThis post announces Facebook's participation in World IPv6 Day, a 24-hour global test flight of IPv6. It explains the necessity of IPv6 due to IPv4 address exhaustion and highlights Facebook's readiness for the transition, anticipating minimal disruption to users. The post also emphasizes the importance of IPv6 for the continued growth and openness of the internet.

Mar 2011Accelerating Network Innovation with the Open Networking FoundationThis post announces Facebook's founding membership in the Open Networking Foundation (ONF) and highlights the importance of Software-Defined Networking (SDN) and the OpenFlow protocol for optimizing network performance. It details how SDN enables direct software control over network resources, leading to benefits like automated failure detection and rerouting, dynamic bandwidth provisioning, and energy savings through traffic-based power management. The post also explains how OpenFlow can improve data center network topologies (fat tree and Clos) to reduce latency and enhance reliability for scaled-out architectures, and mentions Facebook's experimentation with SDN controllers and silicon providers.

Jan 2011HTML5 Games 0.1: Speedy SpritesThis post introduces JSGameBench, a benchmark suite focused on measuring HTML5 game performance, specifically sprite rendering. It details the methodology for testing axis-aligned and rotated sprites at 30fps, the scoring mechanism (geometric mean), and findings on browser performance variance. It also outlines next steps for JSGameBench, including mobile abstractions, networking, prediction/synchronization tools, and reference engines, while also providing feedback to browser developers on areas like mobile auto-rotate control, hardware compositing, and accelerated matrix/transform properties.

Jan 2011World IPv6 Day: Solving the IP Address Chicken-and-Egg ChallengeThis post announces Facebook's participation in World IPv6 Day, a global 24-hour test flight for IPv6. It highlights the critical need for IPv6 due to IPv4 address exhaustion and the challenges of dual-stack connectivity for a small percentage of users. The post contrasts the long-term, end-to-end solution of IPv6 with the less desirable technical alternative of Large Scale NAT (LSN), emphasizing the importance of industry-wide testing and collaboration to ensure robust IPv6 products and reliable dual-stack websites.

Oct 2010Using HTML5 TodayThis post details the early adoption of HTML5 features to improve web application performance and user experience. Specifically, it highlights the use of the Geolocation API for the 'Places' feature, the History API to enhance AJAX loading and URL management, and experiments with Web Storage for caching. It also mentions plans to explore WebSockets for more stable real-time applications, indicating a proactive approach to leveraging new web standards for network efficiency.

Jul 2010Internet CartographyThis post introduces the concept of 'Internet Cartography' by correlating historical telegraph network maps with modern internet topology. It details a method using the open-source Boomerang framework to measure DNS and HTTP latencies from user traffic. The post presents analysis of median and density plots for DNS latencies, and median HTTP latencies, comparing real-world performance against theoretical minimums based on geographical distance. It highlights how factors like ocean width, economic ties, and local geography influence network performance, using examples from Europe, Asia, and South America. The post encourages readers to implement similar measurements from their own websites.

Jun 2010BigPipe: Pipelining web pages for high performanceIntroduced BigPipe, a fundamental redesign of dynamic web page serving that decomposes web pages into pagelets and pipelines them through server and browser execution stages. This enables overlapping server generation time with browser rendering time, significantly reducing user-perceived latency by progressively displaying content. BigPipe was implemented in PHP and JavaScript and demonstrated a reduction in user-perceived latency by half in most browsers compared to traditional single-flush models.

May 2010Protecting Privacy with ReferrersThis post details the technical implementation of a cross-browser compatible redirector designed to remove user IDs from referrer links. It explores various redirection methods (302, Refresh, Meta Refresh, document.location.replace(), anchor click(), 2-way shimmed anchor click) and their browser-specific behaviors regarding referrer information. The chosen solution involves a redirector that successfully redirects users, removes the referrer of the source page, still conveys that a click happened on Facebook.com, and is not an open redirector. A fallback 'Refresh:1' header is included for browsers that do not support JavaScript or have it disabled.

May 2010The Life of a Typeahead QueryIntroduced the architecture for Facebook's typeahead search, detailing the flow from user input to rendered results. This includes browser caching for initial results, AJAX requests, an aggregator service that delegates queries to specialized leaf services (global and graph), result merging, data fetching from memcache/MySQL, and privacy validation. The post also discusses performance optimizations, UI experimentation via A/B testing, and the trilemma of performance, recall, and relevance in real-time search.

Feb 2010Making Facebook 2x FasterThis post details the strategies and technologies implemented to make Facebook twice as fast, focusing on reducing network time and render time. Key contributions include: the definition of Time-to-Interact (TTI) as a primary performance metric; the reduction of average cookie bytes per request by 42%; the development of a reusable component library for HTML and CSS, leading to a 19% reduction in CSS bytes and a 44% reduction in HTML bytes per page; the creation of the Primer JavaScript library, resulting in a 40% decrease in JavaScript bytes per page; and the implementation of the BigPipe system for pipelining Pagelet generation and rendering to reduce TTI.

Aug 2009Every Millisecond CountsThis post details an engineering intern's summer project exploring the impact of site speed on user behavior. Experiments revealed that while overall session time remained relatively constant with slower speeds, page views decreased, indicating that improving site speed allows users to consume more content. The post also resolved a debate on page loading schemes, favoring immediate rendering over a blank page, and highlighted the success of scroll loading for News Feed, which increased usage statistics, particularly for casual users, by enabling the display of more stories without perceived delay.

Apr 2009Scaling Facebook Live Feed for the InaugurationThis post details the strategies used to scale Facebook's live feed for the inauguration event, handling millions of concurrent users and a high request rate. Key technical contributions include: differentiating "Friends" and "Everyone" tab requests, optimizing the "Friends" tab backend by reducing PHP initialization for empty responses, and aggressively caching "Everyone" tab content at multiple layers (memcache and CDN). The use of CDN for dynamic content with short invalidation intervals is highlighted, along with workarounds for caching issues like infinite loops and users falling behind. Extensive load testing and "dark launching" of client-side JavaScript were employed to simulate peak load and test contingency mechanisms.

Aug 2008Scaling OutThis post details the technical challenges and solutions for scaling Facebook's infrastructure by opening a new datacenter on the east coast. Key contributions include: 1. Addressing latency by distributing data centers. 2. Solving cache consistency issues in a multi-datacenter setup with replication lag by modifying MySQL's replication stream to include memcache keys, enabling the secondary database to invalidate relevant cache entries. 3. Implementing traffic routing logic using Layer 7 load balancers and browser cookies to direct write operations to the primary datacenter and manage user experience during replication lag.

What this trail covers