TechPrufer
Get started
Meta
Meta logoMeta · Feature trail

Data Center Energy Efficiency and Optimization

67 posts · Oct 2010 – Jun 2026

What this trail covers

Meta's commitment to optimizing data center energy usage and improving overall efficiency has evolved from initial strategies like improved airflow management and adjusted temperature set points to a fundamental redesign of hardware and data center infrastructure. This includes the development of custom-designed servers, power supplies, racks, and battery backup systems, leading to significant reductions in energy consumption and cost. A key milestone was the formation of the Open Compute Project. This evolution now includes leveraging AI for Scope 3 emission estimates for IT hardware and developing new concrete mixes for data center construction to achieve net-zero emissions.

All Meta trails
  1. Lights Out, Systems On: Validating Instant Power Loss ReadinessThis post introduces 'Instantaneous PowerLoss Storm,' a new testing paradigm for validating data center readiness against zero-notice power loss. It details the defense-in-depth strategies implemented across the data center stack, including solutions for bootstrapping challenges like circular dependencies and the 'boomerang' problem. The post also outlines the tradeoffs made between reliability and growth velocity, defining tolerable impacts, and describes the incremental validation process through exercises in pre-production and production regions, culminating in the simulation of large-scale power loss events.
  2. Capacity Efficiency at Meta: How Unified AI Agents Optimize Performance at HyperscaleThis post introduces Meta's Capacity Efficiency Program, which utilizes a unified AI agent platform to automate the detection and resolution of performance issues across its infrastructure. The platform employs AI agents that encode domain expertise into reusable skills, leveraging standardized tool interfaces for both proactive optimization ('offense') and regression detection/mitigation ('defense'). This has resulted in the recovery of hundreds of megawatts of power and a significant reduction in manual engineering investigation time, enabling the program to scale efficiently.
  3. Investing in Infrastructure: Meta’s Renewed Commitment to jemallocThis post details Meta's renewed commitment to jemalloc, a high-performance memory allocator, emphasizing its role as a foundational component for reliable and performant infrastructure. It highlights efforts to reduce technical debt, modernize the codebase, and collaborate with the open-source community to evolve jemalloc for current and future hardware and workloads, including specific improvements for huge-page allocation, memory efficiency, and AArch64 optimizations.
  4. Open Source Is Good for the EnvironmentThis post introduces a new open methodology for leveraging AI to understand Scope 3 emissions for IT hardware, building upon Meta's existing efforts in data center energy efficiency and optimization. It also highlights the use of AI in developing new concrete mixes for data center construction as part of the company's net-zero emissions goal.
  5. Design for Sustainability: New Design Principles for Reducing IT Hardware EmissionsThis post introduces a new set of technical design principles for IT hardware aimed at reducing emissions and cost. It details strategies for Scope 3 emissions reduction, including modular rack designs (ORv3, ORv3N) with toolless FRUs and flexible power configurations, the benefits and challenges of retrofitting existing rack designs, and the adoption of green steel (produced with renewable energy and recycled content) and recycled aluminum and copper (mandating minimum recycled content). The post emphasizes improving hardware reliability to extend useful life as a key factor in reducing emissions and costs.
  7. How Meta Is Leveraging AI To Improve the Quality of Scope 3 Emission Estimates for IT HardwareThis post details Meta's new methodology for improving Scope 3 emission estimates for IT hardware by leveraging AI. It introduces the use of NLP for identifying similar components to existing Product Carbon Footprints (PCFs) and LLMs (specifically Llama 3.1) for extracting data from heterogeneous sources and creating a standardized taxonomy for IT hardware carbon footprints. The post highlights the collaboration with the OCP PCR workstream to open-source this methodology, aiming to foster industry-wide sustainable manufacturing practices.
  8. OCP Summit 2025: The Open Future of Networking Hardware for AIThis post details Meta's advancements in open networking hardware for AI training clusters, presented at OCP Summit 2025. It introduces the evolution of Disaggregated Scheduled Fabric (DSF) to a dual-stage architecture supporting larger AI clusters, a new Non-Scheduled Fabric (NSF) architecture for massive AI clusters like Prometheus, and new OCP switch platforms like Minipack3N based on NVIDIA's Spectrum-4 ASIC. It also highlights advancements in optical interconnects (2x400G FR4 LITE, 400G/2x400G DR4) and the launch of the Ethernet for Scale-Up Networking (ESUN) initiative, underscoring Meta's leadership in driving open, scalable, and interoperable networking solutions for AI infrastructure.
  9. A case for QLC SSDs in the data centerThis post makes a case for adopting QLC SSDs in Meta's data centers as a new storage tier. It details the technical rationale, comparing QLC to HDDs and TLC SSDs in terms of density, performance, power efficiency, and cost. It discusses hardware considerations, such as form factors (E1.S vs. U.2) and the need for more performant CPU, memory, and network subsystems to leverage QLC's density. It also outlines software adaptations required, including handling higher throughput expectations and tuning rate controllers and I/O schedulers to manage the significant read/write throughput delta characteristic of QLC SSDs. Meta is actively collaborating with partners like Pure Storage and other NAND vendors to integrate QLC solutions.
  10. How Precision Time Protocol handles leap secondsThis post details Meta's approach to handling leap seconds within data centers using Precision Time Protocol (PTP). It contrasts PTP's nanosecond precision with NTP's millisecond precision and explains Meta's 'self-smearing' algorithmic approach via the fbclock library. The post also discusses the trade-offs of this approach, the challenges of integrating PTP with NTP, and advocates for the elimination of leap seconds in favor of UTC for simplified and more precise timekeeping.
  11. OCP Summit 2024: The open future of networking hardware for AIThis post details Meta's latest contributions to the Open Compute Project (OCP) Summit 2024, focusing on next-generation network hardware for AI training clusters. It introduces two new disaggregated network fabrics (DSF) and a new NIC (FBNIC), highlighting their role in enabling more flexible, scalable, and efficient AI infrastructure. The post also showcases specific hardware like the Arista 7700R4 series, Minipack3, and Cisco 8501 switches, along with advancements in optics and the evolution of Meta's FBOSS and OCP-SAI software, all aimed at improving AI data center performance and sustainability through open hardware.
  12. Simulator-based reinforcement learning for data center cooling optimizationThis post details Meta's application of simulator-based reinforcement learning (RL) to optimize data center cooling airflow setpoints. The RL agent operates in a simulated environment, trained on historical data and simulated scenarios, to learn a policy that minimizes energy and water consumption while maintaining environmental specifications. The system achieved an average reduction of 20% in supply fan energy consumption and 4% in water usage in a pilot region. The physics-based simulator is crucial for training the RL model with diverse scenarios and ensuring hardware performance is not affected by temperature fluctuations.
  13. RETINAS: Real-Time Infrastructure Accounting for SustainabilityIntroduced the RETINAS initiative and a new metric: 'real-time server fleet utilization effectiveness.' This metric combines the depreciation of embodied carbon (using financial depreciation concepts applied to server lifespans) with utilization effectiveness (Total resource available / Resource utilized) to provide near real-time insights into the impact of server fleet operations on Scope 3 emissions. The post details the mathematical formulation of the metric and illustrates its application with examples showing how extending server useful life and improving efficiency can lower the metric, enabling data-driven decisions for reducing embodied carbon.
  14. DCPerf: An open source benchmark suite for hyperscale compute applicationsThis post introduces DCPerf, an open-source benchmark suite developed by Meta to accurately represent the diverse workloads found in hyperscale cloud deployments. It addresses the limitations of existing benchmarks by analyzing production workloads and capturing their characteristics, enabling better design and evaluation of server and datacenter hardware and software. DCPerf has been used internally at Meta for product evaluation and capacity planning, and is now being shared with the broader industry to foster collaborative innovation in compute platform designs.
  15. OCP Summit 2022: Open hardware for AI infrastructureThis post announces Meta's contribution of new AI infrastructure hardware designs to the Open Compute Project (OCP) at the OCP Summit 2022. Key contributions include the Grand Teton GPU-based AI platform, an updated Open Rack v3 (ORv3) with improved power and cooling capabilities (including Air-Assisted Liquid Cooling), and the Grand Canyon HDD storage system. It also highlights the evolution of Meta's data center strategy towards supporting large-scale AI workloads and the importance of open collaboration in hardware design.
  16. How thermal simulation helps optimize Meta’s data centersThis post introduces a physics-based thermal simulator, combined with data science, for optimizing Meta's data center cooling and thermal behavior. It details the methodology of building the simulator using Modelica and control Description Language, and demonstrates its predictive accuracy through a case study of a severe winter storm in Texas, achieving a Mean Absolute Error (MAE) of 0.5°F for supply air temperature. The simulator can be used to test control policies with AI or model predictive control, and future research aims to couple it with machine learning methods for real-time optimization and testing new facility designs.
  17. Transparent memory offloading: more memory at a fraction of the cost and powerThis post introduces Transparent Memory Offloading (TMO), Meta's data center solution for addressing the growing memory needs and high cost of DRAM. TMO utilizes cheaper, higher-capacity memory technologies like NVMe SSDs and compressed memory by transparently offloading less frequently accessed memory pages. It employs a new Linux kernel mechanism (PSI) to measure workload pressure and a userspace agent (Senpai) to dynamically adjust offloading, achieving significant memory savings (20-32%) across millions of servers and contributing to reduced cost and power consumption.
  18. OCP Summit 2021: Open networking hardware lays the groundwork for the metaverseThis post details Meta's adoption of open networking hardware and the migration of its data center network to the Open Compute Project (OCP) Switch Abstraction Interface (SAI). It introduces the Wedge 400/400C next-generation Top-of-Rack (TOR) switches, highlighting their increased switching capacity (12.8 Tbps) and performance improvements over previous models. The post also describes the migration of FBOSS (Facebook Open Switching System) to support SAI, enabling easier integration with multiple ASIC vendors. Furthermore, it announces the deployment of 200G optics and next-generation 200G/400G fabric switches like Minipack2 and Arista 7388X5, which offer higher bandwidths (25.6 Tbps) and reduced power consumption per bit, laying the groundwork for future computing platforms like the metaverse.
  19. Open-sourcing a more precise time applianceThis post introduces the open-sourcing of a new Open Compute Time Appliance, including a PCIe Time Card that transforms commodity servers into precise time appliances. This innovation significantly improves the accuracy of Meta's timing infrastructure to 100 microseconds, enhancing data center management and distributed database performance. The post details the technical challenges and solutions involved in building this appliance, emphasizing its advantages over off-the-shelf alternatives in terms of security, reliability, and cost, and highlights the collaborative effort with the Open Compute Project community.
  20. How Facebook keeps its large-scale infrastructure hardware up and runningThis post details Meta's methodologies for maintaining high hardware availability in its large-scale data centers. It introduces systems like MachineChecker and FBAR for hardware failure detection and remediation, Cyborg for lower-level fixes, and a hybrid mechanism for minimizing performance impact during error reporting. Furthermore, it highlights the use of machine learning to predict repairs for undiagnosed/misdiagnosed tickets and an automated root cause analysis tool leveraging Scuba and FP-Growth for log analysis.
  21. FioSynth: A representative I/O benchmark and data visualizer for data center workloadsIntroduced FioSynth, an open-source tool for automating storage workload execution and parsing results. FioSynth synthesizes production I/O traces to simulate diverse Facebook production services, focusing on latency outliers rather than peak IOPS. It uses fio, offers scalable workloads, parses results into CSV, supports client/server mode for parallel execution, optionally collects device health logs for write amplification factor calculation, and includes preconditioning and run cycle definition for repeatability. This tool aids in identifying storage performance inefficiencies and reproducing functional issues.
  22. Pcicrawler: A Python-based command-line interface tool to debug PCI issues at scaleThis post introduces Pcicrawler, a Python-based command-line tool designed to diagnose and debug Peripheral Component Interconnect (PCI) and PCI Express (PCIe) issues at scale. It provides detailed hardware information, visualizes PCI topology, and offers machine-parsable output for automation, contributing to the monitoring, diagnosis, and resolution of hardware-related performance and reliability issues within Meta's data centers.
  23. Accelerometer and SoftSKU: Improving hardware platform performance for diverse microservicesIntroduced SoftSKU, a novel mechanism to optimize server processors for specific microservices by exploiting OS and hardware configuration knobs, achieving up to 7.2% performance improvement without new hardware. Developed Accelerometer, an analytical model to predict hardware acceleration gains with less than 3.7% error rate, aiding in informed hardware investment decisions.
  24. Systems @Scale Tel Aviv 2019 recapThis post discusses advancements in scaling Facebook's data center infrastructure to support billions of users, focusing on technological innovations for reliability, scalability, and efficiency. It touches upon the physical infrastructure aspects of running large-scale services.
  25. Facebook announces next-generation Open Rack frameThis post announces the next-generation Open Rack frame, a collaborative initiative with Microsoft and the Open Compute Project (OCP) community. It details the goals of enabling greater sharing, providing a flexible frame and power infrastructure, supporting advanced thermal solutions (liquid cooling, door-based heat exchangers), and developing scalable power and battery backup solutions. The post highlights the increasing power density driven by AI and ML, the advantages of 48V power delivery over 12V, and the community's development of direct contact liquid loops, door-based heat exchangers, and fully immersed systems. The frame specification is released for community review, with prototypes planned for later in the year.
  26. Reinventing Facebook’s data center networkThis post details the reinvention of Facebook's data center network to address increasing demand and physical constraints. Key technical contributions include the F16 fabric design, which uses 100G CWDM4-OCP optics to achieve 4x capacity with fewer fabric chips and reduced power consumption compared to 400G alternatives. The development of the Minipack switch, a modular and flexible building block consuming 50% less power and space, is also highlighted, along with its contribution to OCP. The post also introduces HGRID, an evolution of Fabric Aggregator, designed to scale aggregation to six buildings per region and flatten the regional network for improved East-West traffic and uplink bandwidth.
  27. Sharing a common form factor for accelerator modulesThis post introduces the OCP Accelerator Module (OAM) as a common form factor specification for hardware accelerators (GPUs, FPGAs, ASICs, etc.) used in AI and high-performance computing. It addresses the limitations of existing form factors like PCIe CEM by providing a more optimized solution for high bandwidth, interconnect flexibility, and power scalability. The OAM specification includes details on power input (12V/48V), TDP support, dimensions, link configurations, and support for various interconnect topologies like hybrid cube mesh and fully connected. Meta has contributed this specification to the Open Compute Project community to foster open collaboration and innovation.
  28. Open-sourcing homomorphic hashing to secure update propagationThis post introduces the application of homomorphic hashing (LtHash) to secure update propagation within Meta's configuration management systems, specifically the Location Aware Distribution system. It details the challenges of maintaining update integrity at scale in distributed systems and presents LtHash as an efficient solution that allows for updates to database hashes without recomputing the entire hash from scratch. The post also announces the open-sourcing of the LtHash implementation in the Folly library.
  29. Rethinking data center design for SingaporeThis post details the design of Meta's first custom-built data center in Singapore, focusing on efficiency and resource conservation. Key innovations include an 11-story vertical design to minimize land footprint, the implementation of the StatePoint Liquid Cooling (SPLC) system for reduced water and energy consumption in humid climates, and a commitment to 100% renewable energy with rooftop solar integration. The facility aims for a PUE of 1.19, significantly lower than the regional average.
  30. Data centers year in reviewThis post details Meta's data center engineering advancements in 2018, including the expansion to 15 locations, the announcement of new sites and expansions, and the commitment to 100% renewable energy. Key technical innovations highlighted are the Fabric Aggregator for flexible network scaling and the StatePoint Liquid Cooling (SPLC) system for energy- and water-efficient cooling, particularly in challenging environments like Singapore, which also features the first multistory data center design. The post also mentions the open-sourcing of StateService and FBOSS.
  31. StatePoint Liquid Cooling system: A new, more efficient way to cool a data centerThis post introduces the StatePoint Liquid Cooling (SPLC) system, a novel indirect evaporative cooling technology developed with Nortek Air Solutions. The SPLC system uses water to cool data hall air, operating in three modes (economizer, adiabatic, and super-evaporative) to optimize water and power consumption based on ambient conditions. It utilizes a membrane separation layer to prevent cross-contamination and is resilient to scale formation. The system reduces water usage by over 20% in hot/humid climates and nearly 90% in cooler climates compared to previous indirect cooling systems, and allows for data center construction in locations previously unsuitable for direct cooling. It also offers flexibility in cooling delivery systems and contributes to a lower Power Usage Effectiveness (PUE).
  32. 2017 Year in review: Data centersThis post details the 2017 advancements in Meta's data center infrastructure, focusing on scaling capacity and improving energy efficiency. Key hardware innovations include Bryce Canyon (storage chassis), Big Basin (AI GPU server), Tioga Pass (CPU server), and Yosemite v2 (multi-node compute platform with hot-swapping). A significant migration from InnoDB to MyRocks for the user database is also highlighted, which halved server and storage needs for the social graph. The post also reiterates the commitment to 100% clean and renewable energy for new data centers.
  33. Hardware Analytics and Lifecycle Optimization (HALO) at FacebookThis post introduces the Hardware Analytics and Lifecycle Optimization (HALO) system at Facebook. HALO is designed to monitor, process, and visualize data from millions of hardware components to improve global hardware health and inform future hardware design. It leverages Lumberjack for data collection (SMART attributes, RAID logs, network interface logs, etc.), integrates with asset and ticketing systems, and uses Facebook's Operational Data Store (ODS). Data processing involves analyzing failure rates and error categories at a fleet level, with tools like Attribot for identifying known hardware issues and Systemic Issue Detection (SID) for identifying emerging systemic hardware problems. The outcomes include improved global hardware health through cross-functional remediation efforts with vendors and internal teams, and design improvements based on field experience, such as toolless serviceability and airflow design adjustments.
  34. OCP Summit 2017 — Facebook news recapThis post details Meta's contributions to the Open Compute Project (OCP) Summit 2017, focusing on new hardware designs and networking updates. Key contributions include Big Basin (next-generation GPU server for AI model training), Bryce Canyon (high-density storage platform), Tioga Pass (successor to Leopard compute platform with OpenBMC), and Yosemite v2 (updated multi-node compute platform with hot-service capabilities). Networking updates include the CWDM4-OCP specification for 100G optical transceivers, the Backpack modular switch platform, and the Wedge 100S top-of-rack switch with enhanced boot security and support for SAI and SONiC.
  35. The end-to-end refresh of our server hardware fleetThis post announces an end-to-end refresh of Meta's server hardware fleet, introducing four new designs: Bryce Canyon (storage chassis), Big Basin (GPU server for AI training), Tioga Pass (compute server), and Yosemite v2 (multi-node compute platform). These designs leverage the Open Compute Project standards and incorporate improvements in density, performance, thermal efficiency, and serviceability. Key innovations include higher HDD density in Bryce Canyon, increased training throughput and memory in Big Basin, enhanced PCIe bandwidth and networking in Tioga Pass, and hot-swappable server cards in Yosemite v2. The designs also integrate OpenBMC for system management and are compatible with Open Rack v2.
  36. Introducing Bryce Canyon: Our next-generation storage platformThis post introduces Bryce Canyon, Meta's next-generation storage platform, designed for high-density storage of photos and videos. It builds upon previous OCP contributions like Open Vault and Honey Badger, offering 20% higher HDD density, improved thermal and power efficiency, and flexible configurations (single/dual server, JBOD). Bryce Canyon supports powerful processors and more memory, leverages Mono Lake CPU modules for efficiency, and utilizes OCP mezzanine NICs. Its modular design allows for future upgrades, and it emphasizes toolless servicing for simplified data center operations.
  37. Designing 100G optical connectionsThis post details the development and deployment of 100G single-mode optical transceivers, a crucial step in upgrading data center network speeds. It highlights the challenges of moving to higher data rates, the decision to use single-mode fiber for future-proofing, and the optimization of transceiver specifications (CWDM4-OCP) to reduce power consumption and cost. The contribution to the Open Compute Project (OCP) is also emphasized, making this technology accessible to the wider industry.
  38. The growing ecosystem around open networking hardwareThis post details the growing ecosystem around Meta's open networking hardware initiatives, such as Wedge 100 and Backpack. It highlights how various companies are building software and hardware solutions that integrate with these open designs, fostering collaboration and accelerating innovation in data center networking. This expands the scope of the feature thread to include the impact of open hardware on the broader networking industry and the collaborative development of infrastructure.
  39. Sustainable materials in the data centerThis post details the adoption and implementation of natural fiber-filled polypropylene (NFFPP) as a sustainable alternative to PCABS in data center hardware. It covers the material properties of NFFPP (including flammability ratings and shrinkage rates), design considerations for incorporating fiber-filled composites, and specific adjustments to the molding process. The post highlights the successful conversion of several Open Compute Project hardware components (Lightning, Honey Badger, Yosemite, Open Rack, Knox, Leopard) to NFFPP, demonstrating a reduction in upstream carbon footprint by over 50% compared to PCABS.
  40. Introducing Backpack: Our second-generation modular open switchThis post introduces Backpack, Meta's second-generation modular open switch platform, designed to support the migration to 100G data centers. It highlights the hardware design challenges, including improved cooling systems for higher power ASICs and optics, and a disaggregated architecture for scalability and thermal performance. The post also details the software stack (FBOSS, OpenBMC), testing methodologies, and the contribution of Backpack's specifications to the Open Compute Project, emphasizing its role in building a more efficient and scalable data center infrastructure.
  41. Wedge 100: More open and versatile than everThis post details the Wedge 100, a second-generation top-of-rack network switch designed for 100G data centers. It highlights the hardware updates for improved serviceability, thermal management, and Open Rack compatibility, as well as software updates to the FBOSS stack to support the new platform and enhance operational flexibility. The post also announces the acceptance of the Wedge 100 specification into the Open Compute Project and discusses the broader hardware and software ecosystem, including commercial availability and third-party software integrations.
  42. Facebook in Los Lunas: Our newest data centerAnnounces the location and planned opening of Meta's seventh data center in Los Lunas, New Mexico. Highlights that the facility will utilize the latest OCP compute, storage, and networking technologies and be powered by 100-percent renewable energy, emphasizing its efficiency and sustainability.
  43. Facebook opens lab to others to validate infrastructure softwareThis post announces the creation of a lab space at Meta's Menlo Park headquarters specifically for vendors to test their software on open hardware. This initiative aims to address the challenge of ensuring compatibility between commercial software and customized open hardware, which is a barrier for some companies adopting open hardware strategies. The lab has already seen Canonical and Red Hat validate their software packages (MAAS, Juju, Gluster Storage, Ansible, CloudForms, OpenStack) on Facebook's open hardware components (Leopard servers, Honey Badger storage, Knox JBODs), with most software working out of the box. The goal is to accelerate industry adoption of open hardware by providing a systematic way to validate compatibility.
  44. Inside Facebook’s hardware labs: Moving faster with more collaborationThis post details the creation and capabilities of 'Area 404', a new 22,000-square-foot collaborative hardware lab. It highlights the lab's state-of-the-art machine tools (9-axis mill-turn lathe, 5-axis vertical milling machine, 5-axis water jet, sheet metal shear and folder, CNC fabric cutter, CMM, electron microscope, CT scanner) and their application in accelerating hardware development cycles for infrastructure, Connectivity Lab, Oculus, and Building 8 teams. The post emphasizes the shift from isolated labs to a collaborative environment, reducing iteration time from weeks to days and enabling in-house modeling, prototyping, and failure analysis for projects like custom server racks, optical detectors, and VR camera rigs.
  45. Open Compute Project U.S. Summit 2016 — Facebook news recapThis post details Facebook's contributions and collaborations announced at the Open Compute Project U.S. Summit 2016. Key contributions include the open-sourcing of Lightning (an NVMe-based storage platform), OpenBMC (board management software supporting storage), Yosemite (an open server design) and Mono Lake (a server board), Wedge 100 (a 32x100G network switch) and 6-pack (an open modular switch platform), and Big Sur (an open AI hardware platform). Facebook also announced collaborations on 48V rack power distribution with Google and on optical interconnect standards (100G CWDM4) with other industry players. The post highlights the benefits of open hardware and disaggregation for scalability, flexibility, and faster innovation in data center technology, enabling new shared experiences and supporting AI research.
  46. Facebook’s new front-end server design delivers on performance without sucking up powerThis post details the redesign of Facebook's web servers to improve performance per watt. Key contributions include a collaborative effort with Intel to develop a new one-processor Xeon-D CPU (Broadwell-D) with a lower TDP (65W), and a redesigned server infrastructure (Mono Lake boards and Yosemite sleds) that doubles the number of CPUs per rack within the existing 11kW power budget. This approach eliminates NUMA issues and improves overall data center efficiency by achieving higher compute density without exceeding power constraints. The post also highlights the potential for LLC cache partitioning to further optimize memory bandwidth and performance.
  47. Introducing Lightning: A flexible NVMe JBOFThis post introduces Lightning, Meta's first NVMe JBOF (Just a Bunch of Flash) solution designed for disaggregated storage. It details the hardware components, including a PCIe retimer card, PCIe Expansion Board (PEB), and PCIe Drive Plane Board (PDPB), enabling end-to-end PCIe Gen 3 connectivity. The design addresses challenges like hot-plug support, management, signal integrity, external cabling, and power consumption, aiming to provide a scalable and flexible flash building block for data centers. The solution is contributed to the Open Compute Project.
  48. OpenBMC: One board management software for all hardware at FacebookThis post details the extension of OpenBMC to support Facebook's NVMe-based storage platform, Lightning. It describes the unique interface requirements for managing Lightning's BMC (USB, I2C, PCIe) due to the absence of a traditional NIC, and introduces new features like 'fscd' for fan control, 'log-util' for capturing secondary device logs, and 'flash-util' for reading SSD sideband status and managing the PCIe switch. The post highlights how this brings uniform board management across all Facebook data center hardware, simplifying provisioning and improving fault analysis.
  49. Opening designs for 6-pack and Wedge 100This post details the opening of designs for the 6-pack and Wedge 100 network switches to the Open Compute Project (OCP). It highlights the evolution of Meta's data center networking from hierarchical systems to a data center fabric, with Wedge and 6-pack as key components. The post emphasizes the modularity, 100G connectivity support, and extensive hardware and network-level testing performed, showcasing a commitment to open hardware and collaborative development in advancing data center infrastructure efficiency.
  50. Facebook in Ireland: Our newest data centerThis post announces the construction of Meta's new data center in Clonee, Ireland. It highlights the facility's commitment to 100% renewable energy and its utilization of advanced Open Compute Project (OCP) hardware, specifically mentioning Yosemite for compute and fabric, Wedge, and 6-pack for the network stack. The data center is positioned as a central component of Facebook's global infrastructure.
  51. Open networking advances with Wedge and FBOSSThis post details the advancement of Meta's open networking initiatives with the introduction and deployment of the Wedge network switch and FBOSS software. It highlights the disaggregation of networking hardware and software, enabling customization and faster innovation. The post discusses the operationalization of Wedge and FBOSS at scale, including custom deployment tools, monitoring strategies, and the implementation of nonstop forwarding. It also shares lessons learned from treating network switches like servers, focusing on protecting the CPU and control plane, and outlines future plans for scaling open networking solutions with Wedge 100 and 6-pack.
  52. OpenBMC for server: Porting and supporting new features for “Yosemite”This post details the porting and enhancement of OpenBMC for the 'Yosemite' multi-node server platform. Key contributions include removing the insecure RMCP+ protocol, adding SSH and REST API support for improved management and security, implementing IPMI FRUID support, sensor monitoring, multi-node support for up to four server boards, Ethernet driver enhancements with NCSI, I2C driver secondary support, an IPMB framework for BMC and Bridge-IC communication, IPMI SDR support with caching, discrete signal monitoring, and new utilities for platform management. These advancements enhance the manageability, security, and efficiency of Meta's server infrastructure.
  53. Using ISC Kea DHCP in our data centersThis post details Meta's migration of its data center DHCP infrastructure from ISC dhcpd to ISC Kea. This migration was driven by the need for faster server provisioning and reduced downtime, especially at scale. By leveraging Kea's extensibility and hook points, Meta was able to create a stateless DHCP service that dynamically fetches configuration from its inventory system, simplifying deployment and maintenance, and ultimately contributing to faster hardware bootstrapping and repair times.
  54. Facebook in Fort Worth: Our newest data centerThis post announces the construction of Meta's fifth U.S. data center in Fort Worth, Texas. It highlights that the facility will feature the latest Open Compute server, storage, and network designs, and will be powered by 100 percent renewable energy through a new wind deal. This contributes to the ongoing evolution of Meta's data center infrastructure with a focus on advanced designs and sustainable energy.
  55. Open Compute Project U.S. Summit 2015 – Facebook News RecapThis post announces Facebook's contributions and announcements at the Open Compute Project U.S. Summit 2015, highlighting the open-sourcing of all major physical components of their data center stack. Key announcements include the introduction of 'Yosemite', a modular chassis for high-powered microservers; the proposal of the 'Wedge' network switch spec to the OCP Foundation and its availability as a product package; the release of 'OpenBMC', an open software framework for system management; and the availability of the 'FBOSS Agent' for programming the Wedge switch's ASIC. The post also reiterates significant cost and energy savings achieved through OCP and related efficiency work.
  56. Introducing “Yosemite”: the first open source modular chassis for high-powered microserversIntroduced "Yosemite," an open-source modular chassis design for high-powered microservers. Yosemite utilizes System-on-Chip (SoC) processor cards (up to 65W TDP) within a pluggable module format, supporting multiple memory channels, local SSDs, and a management controller. The system is designed for scale-out architectures, accommodating four SoC cards in a 400W total power chassis and integrating with Open Rack for high density. A key innovation is a multi-host network interconnect card that provides shared data and management traffic over a single Ethernet port, improving external connectivity simplicity and efficiency.
  57. Facebook Open Switching System (“FBOSS”) and Wedge in the openThis post announces the open-sourcing of FBOSS (Facebook Open Switching System) and Wedge, a top-of-rack switch specification. It details the disaggregation of network hardware and software, making switches more like servers for easier management and development. FBOSS is presented as a set of applications running on Linux, rather than a proprietary OS, and includes the FBOSS agent for programming ASICs and OpenBMC for system management. The contribution of Wedge to the Open Compute Project (OCP) signifies a move towards open networking standards and hardware, aiming to foster community development and broader adoption of open-source networking solutions. This directly supports the broader goal of optimizing data center infrastructure through open and flexible solutions.
  58. Introducing “OpenBMC”: an open software framework for next-generation system managementThis post introduces OpenBMC, an open software framework for next-generation system management developed by Meta. It addresses the limitations of closed BMC software stacks by enabling rapid iteration, customization, and collaboration. OpenBMC is built as a flexible Linux distribution using the Yocto Project and has been deployed with Meta's Wedge switch. The post highlights the potential for community contributions to the common, SoC, and board layers, fostering innovation in system management applications and hardware support, and aligning with the Open Compute Project's goals.
  59. Serving Facebook Multifeed: Efficiency, performance gains through redesignThis post introduces the concept of 'disaggregation' as a strategy for optimizing data center infrastructure and improving efficiency and performance. It details how this concept was applied to redesign the Multifeed system, separating the CPU-intensive Aggregator component from the memory-intensive Leaf component. This disaggregation led to significant improvements in hardware utilization, scalability, performance (10% latency reduction), and reliability, with a notable 40% efficiency improvement in memory and CPU consumption. The post also highlights the potential for disaggregation in other services like Search and Operational Analytics.
  60. Favorite Hacks of 2014This post details the development of Autoscale, a power-efficient load balancing system created during a hackathon. Autoscale reduces CPU consumption through adaptable decision logic and smarter distribution, leading to significant power savings (10-15% on average for web clusters). It also mentions drone hacks using Android SDKs for experimental purposes and the @Scale app built with Parse for event navigation. The osquery tool, an open-source infrastructure insight tool, was also launched and enhanced through a hackathon challenge.
  61. Introducing data center fabric, the next-generation Facebook data center networkThis post introduces the 'data center fabric,' a next-generation network architecture designed to replace traditional cluster-based designs. It details a disaggregated approach using server pods and fabric switches, connected via spine switches, to create a unified, high-performance network. The post highlights the benefits of this architecture, including rapid deployment, scalability, non-oversubscribed performance, and operational efficiency, achieved through standard BGP4 routing, layer3 connectivity, and a hybrid control plane. This represents a significant evolution in Meta's data center infrastructure, focusing on network performance and scalability to support exponential growth in machine-to-machine traffic.
  62. Making Facebook’s software infrastructure more energy efficient with AutoscaleThis post introduces Autoscale, a power-efficient load balancing system for Facebook's web clusters. Autoscale dynamically adjusts the active server pool to ensure each active server maintains at least medium-level CPU utilization, thereby avoiding inefficient low-utilization states. It uses a feedback loop control system with a PI controller and models the relationship between CPU utilization and RPS based on experimental data. Deployed to production, Autoscale achieved an average of 10-15% power savings over a 24-hour cycle, with up to 27% savings during low-workload periods.
  63. Introducing “Wedge” and “FBOSS,” the next steps toward a disaggregated networkThis post introduces 'Wedge,' a disaggregated top-of-rack network switch, and 'FBOSS,' its Linux-based operating system. These projects represent a significant step in disaggregating network hardware and software, applying the same modular and open-source principles pioneered by the Open Compute Project (OCP) to networking. Wedge offers a modular hardware design leveraging server-like components for greater flexibility and integration into existing fleet management systems. FBOSS provides a Linux-based OS that allows network devices to be managed and controlled similarly to servers, enhancing visibility, automation, and control, and enabling faster implementation of forwarding software and hybrid control logic for optimized link utilization and faster failure recovery. This initiative aims to make network operations more efficient and scalable, aligning with Meta's broader goals of disaggregation and open hardware.
  64. Reflections on the Open Compute SummitThis post details the first Open Compute Summit, highlighting new initiatives in server hardware design that double compute density with dual-motherboard designs supporting next-generation processors. It also introduces a new storage server platform with a flexible compute-to-storage ratio, capable of supporting 50 hard drives per node. The post provides an update on the Prineville data center's energy usage (PUE of 1.077) and discusses plans for the Forest City, North Carolina data center, including increased inlet temperatures and humidity to further reduce environmental impact and hardware requirements. A significant focus is placed on building the Open Compute community through earlier design collaboration, the announcement of a non-profit foundation, and sharing external feedback from companies like Rackspace, Goldman Sachs, AMD, and Microsoft.
  65. How Project Triforce Prepared our Software Stack for PrinevilleThis post details 'Project Triforce,' an internal initiative to simulate a third data center region using an existing cluster. This was crucial for testing and preparing Facebook's complex software stack for the new, custom-built data center in Prineville, Oregon, which was part of a broader strategy to build data centers from the ground up and share hardware designs through the Open Compute Project. It highlights the challenges of scaling infrastructure, software complexity, new configurations (like Flashcache with MySQL), and the need for rapid deployment, solved by tools like Kobold for automated provisioning.
  66. Designing a Very Efficient Data CenterThis post details the design innovations of Meta's Prineville data center, a model for the Open Compute Project. Key contributions include achieving a PUE of 1.07 and WUE of 0.31 liters/kWH, reducing CapEx by 45% and OpEx. Innovations focused on elimination: centralized UPS (replaced with server cabinet-level 48VDC UPS), PDUs (replaced with 277VAC distribution), chillers (replaced with outside air evaporative cooling), and air distribution ductwork. Electrical efficiency was improved by reducing power conversion losses from 21-27% to 7.5%. Cooling innovations leveraged the high desert climate for evaporative cooling and ductless air distribution, with waste heat recirculation.
  67. Inside the Open Compute Project ServerThis post details the design and specifications of the Open Compute Project server, a custom-engineered hardware solution developed by Facebook engineers and industry partners. It covers the design of motherboards (for AMD and Intel CPUs), a highly efficient power supply (94.5% efficiency) with novel backup system (48VDC input), a 'vanity-free' chassis designed for ease of service and reduced weight, a 'triplet' rack designed for efficient server deployment and networking port utilization, a battery cabinet offering a more efficient and cost-effective alternative to traditional UPS systems, and a thermal design utilizing larger heat sinks and fans for reduced energy consumption. The post highlights the collaborative nature of the Open Compute Project and its goal of fostering industry-wide adoption of efficient hardware practices.
  68. Building Efficient Data Centers with the Open Compute ProjectThis post details the design and construction of Facebook's first custom-built data center, focusing on energy efficiency and cost reduction. Key innovations include a 480-volt electrical distribution system, removal of non-essential server components, reuse of hot aisle air for heating, and elimination of central UPS. These changes resulted in a 38% reduction in energy usage and a 24% cost decrease compared to existing facilities. The post also announces the formation of the Open Compute Project to share these hardware and data center specifications with the industry.
  69. New Cooling Strategies for Greater Data Center Energy EfficiencyThis post details specific strategies implemented at a second Facebook data center to improve energy efficiency, resulting in annual electricity savings of 2,876,360 kWh and a reduction in annual demand by 329 kW. Key strategies included extending air-side economizer use, elevating rack inlet temperatures from 51°F to 67°F, reducing excess air supply by replacing unnecessary perforated floor tiles with solid ones, implementing cold aisle containment, and optimizing server fan speeds. These measures significantly increased free cooling hours and reduced mechanical cooling demand.
  70. Optimizing Data Center Energy UsageThis post details specific improvements implemented in Facebook's data centers to optimize energy usage. These include: 1. Improving airflow distribution through cold aisle containment, skirting around PDUs, and sealing cable cut-outs. 2. Reducing cooling levels by optimizing server fan speed control algorithms and shutting down 15 CRAH units. 3. Raising rack inlet temperatures from 72 to 81 degrees Fahrenheit and chilled water temperatures from 44 to 52 degrees Fahrenheit. These measures resulted in annual savings of almost 2.5 million kWh and a reduction in energy demand by 276 kW.