AI Research and Development

8/3/2026

This post details Meta's advancements in optimizing the training efficiency of its LLM-scale ads foundation model, GEM. Key contributions include: 1. Co-designing compute and scaling efficiency innovations to double end-to-end training efficiency to 20-25% MFU and scale training FLOPs 4x in 12 months. 2. Developing a customized recommendation kernel library, including Jagged Flash Attention (JFA) to eliminate padding waste on jagged inputs and improve backward pass efficiency, and Generalized Dot-Product Attention (GDPA) to unify and accelerate diverse, asymmetric attention modules. 3. Implementing mixed ultra-low precision training (MXFP8 attention and MLP) optimized for recommendation workloads. 4. Employing topology-aware 5D parallelism with SM-free collectives (2D FSDP + Expert Parallelism for dense parameters, Fully Sharded 2D Model Parallelism for sparse parameters) co-designed with Meta's multi-tiered network hierarchy to reduce communication overhead.

7/15/2026

This post introduces Hierarchical Interest Representation, a new upstream representation layer for Meta Ads deep funnel optimization. It details the research area, its technical innovations (transformer-based graph learning, bias-aware attention, self-supervised cross-view distillation), and its application in connecting users' inferred interests with advertiser offerings. The post highlights the system's ability to blend real-world knowledge with engagement signals, process multimodal content via LLMs, and generate universal embeddings and interest tokens. It also outlines the technical challenges of user inferred signal dynamics, large networks with sparse connections, and long-range global relationships, and describes the architecture including the enriched engagement graph and hierarchical encoder with bias composition and attention kernel.

7/1/2026

This post details the evolution of Meta's BLOB-storage architecture to meet the demands of large-scale AI workloads. Key technical contributions include: a unified metadata schema backed by ZippyDB for O(1) lookups, elimination of dataplane proxy with a fat client SDK for direct streaming from storage servers, regional deployment of BLOB-storage colocated with GPUs, implementation of a distributed data cache leveraging spare GPU host memory (reusing Owl subsystem components), and a readplan metadata cache. Protocol optimizations include hedged reads for laggards and dynamic concurrency control on the client SDK to manage egress spikes. The architecture aims to maximize GPU utilization and research velocity by minimizing storage bottlenecks and data movement.

5/13/2026

This post details the engineering effort behind the 'Friend Bubbles' feature in Reels, highlighting the evolution of the underlying machine learning model. It emphasizes the challenges of building social discovery features that scale to billions of users, including considerations for cross-platform user behavior (iOS vs. Android) and the iterative process of refining ML models to achieve desired user engagement.

4/6/2026

This post details the development of a pre-compute engine using specialized AI agents to systematically map tribal knowledge within Meta's large-scale data processing pipelines. This engine creates concise context files that act as structured navigation guides for AI agents, significantly improving their understanding and efficiency when working with proprietary codebases. The system addresses the challenge of AI agents lacking context in complex, multi-repository, multi-language code by documenting non-obvious patterns and cross-module dependencies, leading to reduced AI tool calls and faster task completion.

4/2/2026

Introduced KernelEvolve, an agentic kernel authoring system used by Meta's Ranking Engineer Agent to optimize low-level infrastructure for AI models. KernelEvolve treats kernel optimization as a search problem, autonomously generating and optimizing production-grade kernels for heterogeneous hardware (NVIDIA GPUs, AMD GPUs, MTIA silicon, CPUs). It achieves faster development cycles (weeks to hours) and significant performance improvements (over 60% inference throughput on NVIDIA GPUs, over 25% training throughput on MTIA chips) by exploring hundreds of alternative kernel implementations.

3/31/2026

This post introduces the Meta Adaptive Ranking Model, a significant advancement in scaling AI recommendation systems for ads to LLM-scale complexity. It addresses the inference trilemma by employing intelligent request routing, inference-efficient model scaling (request-centric architecture, Request-Oriented Optimization, Wukong Turbo), deep model-system co-design (selective FP8 quantization, hardware-aware graph and kernel specialization), and a reimagined serving infrastructure (trillion-parameter scale, multi-card GPU serving). This innovation has resulted in a +3% increase in ad conversions and +5% increase in ad click-through rate on Instagram.

3/30/2026

This post details the release of BOxCrete, a new AI model for designing concrete mixes, and the foundational data used to develop it. It highlights the application of Bayesian Optimization and adaptive experimentation to address challenges in concrete mix design, aiming to increase the use of American-produced cement and concrete. The post also discusses partnerships with industry leaders and the open-sourcing of the AI framework.

3/17/2026

Introduced the Ranking Engineer Agent (REA), an autonomous AI agent designed to drive the end-to-end ML lifecycle for ads ranking models. REA addresses long-horizon, asynchronous workflow autonomy via a hibernate-and-wake mechanism, high-quality/diverse hypothesis generation using a dual-source engine (historical insights database and ML research agent), and resilient execution within real-world constraints through a three-phase planning framework (Validation, Combination, Exploitation). REA's architecture comprises REA Planner and REA Executor, supported by a Skill, Knowledge and Tool System. In its first production rollout, REA achieved 2x model accuracy and 5x engineering output.

2/24/2026

This post introduces RCCLX, an open-sourced enhanced version of RCCL for AMD platforms, integrated with Torchcomms. It details two key innovations: Direct Data Access (DDA) for lightweight intra-node collectives, improving AllReduce latency by allowing direct memory access and reducing latency from O(N) to O(1) or a constant factor, and Low Precision Collectives (LP Collectives) which use FP8 quantization for up to 4:1 compression to reduce communication overhead and improve scalability for large message sizes. DDA shows 10-50% performance improvements on AMD MI300X GPUs for decode and 10-30% for prefill, resulting in ~10% reduction in time-to-incremental-token. LP Collectives demonstrate significant speedups for FP32 and notable improvements for BF16, with observed ~7% increase in throughput and ~9-10% decrease in latency for E2E inference workloads, while maintaining acceptable numerical accuracy.

2/9/2026

This post details the crucial role of Backend Aggregation (BAG) in building gigawatt-scale AI clusters like Prometheus. It explains how BAG interconnects thousands of GPUs across multiple data centers and regions by acting as a centralized Ethernet-based super spine network layer. The post elaborates on BAG's implementation, including its connection to different network fabrics (DSF and NSF), inter-BAG topologies (planar and spread), hardware and routing strategies (Jericho3 ASICs, eBGP with UCMP), and design considerations for resilience and long cable distances.

1/14/2026

Introduced the User True Interest Survey (UTIS) model for Facebook Reels, which directly incorporates user feedback from surveys to improve personalized video recommendations. This model moves beyond traditional engagement signals (likes, watch time) to capture 'true interests', addressing challenges like sparse user data and bias. The UTIS model was integrated into both late-stage and early-stage ranking systems, demonstrating significant offline improvements in accuracy, precision, and recall, and online improvements in user engagement and retention metrics.

11/21/2025

This post introduces Zoomer, Meta's comprehensive, automated debugging and optimization platform for AI training and inference workloads. Zoomer provides deep performance insights to improve energy savings, accelerate workflows, and enhance efficiency across Meta's AI infrastructure, delivering training time reductions and significant QPS improvements. It details Zoomer's architecture, profiling mechanisms, data capture, distributed analysis pipeline, and specialized workload support for training, inference, and GenAI.

11/10/2025

This post introduces Meta's Generative Ads Model (GEM), a novel foundation model for ads recommendation systems. GEM features an LLM-inspired architecture with innovations in model scaling (4x efficiency), post-training knowledge transfer (2x effectiveness of standard distillation), and training infrastructure (23x effective training FLOPs, 1.43x MFU). Key technical contributions include: scalable architecture with customized attention mechanisms for sequence and non-sequence features, enhanced Wukong architecture for non-sequence feature interaction modeling, pyramid-parallel structure for offline sequence feature modeling, InterFormer for cross-feature learning, and multi-domain learning for surface-specific optimization. Post-training techniques include Student Adapter for knowledge distillation, representation learning, and parameter sharing. The training stack was re-engineered for massive GPU utilization and efficiency.

10/17/2025

This post details Meta's advancements in scaling LLM inference by implementing and optimizing tensor parallelism (TP), context parallelism (CP), and expert parallelism (EP). It introduces direct data access (DDA) algorithms (flat and tree) to reduce allreduce latency in TP, achieving significant speedups over NCCL/RCCL. For CP, it describes 'ring attention' variants (Pass-KV, Pass-Q) enabling efficient long-context processing, achieving sub-minute inference for 1M tokens. For EP, it discusses optimizations for MoE models, including dynamic and persistent all-to-all communication. The post also outlines future directions like N-D parallelism and disaggregated inference tiers.

9/30/2025

This post details the development and application of Meta's Automated Compliance Hardening (ACH) tool, which uses Large Language Models (LLMs) to revolutionize software testing and compliance. It explains how LLMs overcome the traditional barriers to mutation testing, enabling scalable, realistic mutant generation and the detection of equivalent mutants. The post highlights ACH's success in generating privacy-relevant tests and its deployment across various Meta platforms, emphasizing the acceleration of developer velocity and the creation of a safer online ecosystem.

9/29/2025

This post details Meta's infrastructure evolution driven by the advent of AI, highlighting the shift from scaling traditional software and physical infrastructure to building specialized AI clusters. It explains the emergence of GPUs for personalized recommendations and the subsequent dramatic scaling of compute capacity required for Large Language Models (LLMs). The post also emphasizes Meta's commitment to open source and open standards in hardware and silicon for AI infrastructure.

9/26/2025

This post highlights the critical role of networking infrastructure in enabling Meta's large-scale AI advances. It details the evolution of AI networking, emphasizing the need for robust infrastructure to support massive AI build-outs, including gigawatt-scale clusters and subsea fiber cables. The post discusses the rapid evolution of AI workloads beyond foundational training to include mixture-of-experts, reasoning models, and inference, each with distinct network requirements. It underscores the network's function as an abstraction layer, making the infrastructure appear as a single, gigantic GPU, and the necessity of co-designing the network with the AI stack for optimal performance and reliability. The post also emphasizes continuous innovation and optionality in networking to adapt to future AI developments.

8/14/2025

This post details the development of an agentic system for Meta's data warehouse to enhance data access productivity and security. It introduces user agents and owner agents, each composed of specialized sub-agents, to manage data access requests and security operations. The system leverages the hierarchical structure of the data warehouse for agent communication and implements context and intention management. A key use case, 'partial data preview,' is described, orchestrating context, granular query-level access control, data-access budgets, and rule-based risk management.

7/28/2025

This post details the adoption and impact of ExecuTorch, Meta's open-source PyTorch inference framework for edge devices, across Instagram, WhatsApp, Messenger, and Facebook. It highlights how ExecuTorch has accelerated on-device ML by improving model performance, reducing latency, and enhancing user privacy. Specific examples include migrating Instagram's Cutouts feature to ExecuTorch for faster sticker generation, optimizing WhatsApp's bandwidth estimation models for improved video call quality and reduced ANRs, enabling end-to-end encryption on Messenger by moving server-side models like language identification on-device, and enhancing Facebook Stories with background music recommendations powered by the SceneX model. The post also mentions the integration of fuzzing tests for enhanced security in WhatsApp and the ongoing migration of other key WhatsApp and Messenger models to ExecuTorch.

7/22/2025

This post details Meta's methodologies for detecting and mitigating hardware faults, particularly silent data corruptions (SDCs), within its AI infrastructure. It highlights the unique challenges SDCs pose for AI training (e.g., NaN propagation, corrupted gradient variance) and inference workloads, and introduces novel detection mechanisms like Fleetscanner, Ripple, and Hardware Sentinel to ensure the reliability of AI hardware at scale.

7/16/2025

Introduces an open-source AI tool for designing sustainable concrete mixes using Bayesian optimization (BoTorch, Ax). The tool optimizes for strength, sustainability, and curing speed, addressing challenges in traditional concrete formulation. It was successfully deployed at a Meta data center construction site.

5/21/2025

This post details Instagram's journey to scale its recommendation system to over 1000 ML models. It highlights the infrastructure challenges faced, including model discovery, safe release, and health monitoring. The post introduces solutions like a model registry for a single source of truth, automated model launch tooling to reduce time-to-launch from days to hours, and the operationalization of 'model stability' as a key metric for prediction accuracy. It also elaborates on the model registry's structure, including model types and criticality tiers, and its integration with Meta's configuration and monitoring tools.

3/4/2025

This post details the engineering challenges and advancements in building multimodal AI for Ray-Ban Meta glasses. It highlights the development of foundational models capable of processing multiple input types (speech, text, images) for wearable devices, referencing research like AnyMAL and discussing unique challenges in AI glasses and scaling AI for billions of users. The post also touches upon acoustic modalities beyond speech and the processing of moving images.

12/2/2024

This post introduces Meta Andromeda, a next-generation personalized ads retrieval engine that represents a significant advancement in AI for ad recommendation. It details how Andromeda pushes the boundaries of AI for retrieval by leveraging specialized hardware (NVIDIA Grace Hopper Superchip and MTIA) and innovations in ML model architecture, feature representation, learning algorithms, indexing, and inference paradigms. Key contributions include achieving a +6% recall improvement and +8% ads quality improvement, enabling hierarchical indexing to support exponential ad creative growth from Advantage+ automation, and enhancing AI development efficiency by minimizing system complexity. Andromeda represents a new paradigm in personalized ads retrieval, moving beyond previous limitations with a state-of-the-art deep neural network, hierarchical indexing, model elasticity, and an optimized retrieval model that significantly improves performance and ROI.

11/19/2024

This post introduces a significant evolution in Meta's AI capabilities by detailing a paradigm shift in ads recommendation systems. It moves from traditional Deep Learning Recommendation Models (DLRMs) that relied on human-engineered features to a new system based on sequence learning. This involves event-based learning and the development of new sequence learning architectures that leverage sequential information from user engagement and conversion events. The post highlights the limitations of DLRMs in capturing sequential and granular information, and outlines the technical advancements, including event-based features (EBFs) and custom transformer architectures, implemented to achieve this shift. It also discusses the scaling challenges and solutions for both sequence learning architectures and the richness of event sequences, ultimately leading to improved ad relevance and performance.

10/15/2024

This post introduces Meta's open AI hardware vision, detailing new initiatives like the Catalina high-powered rack for AI workloads, the expansion of the Grand Teton platform to support AMD accelerators, and the Disaggregated Scheduled Fabric (DSF) for next-generation AI clusters. It highlights the importance of open hardware for scaling AI infrastructure, discussing advancements in power capabilities (140kW for ORv3), liquid cooling, modular design, and the use of open standards like OCP-SAI and FBOSS for networking. The post also emphasizes the collaboration with Microsoft on disaggregated power racks like Mount Diablo.

10/3/2024

This post details the open-sourcing of training data and sample code for Meta's AI-powered population maps. It explains the methodology, which uses computer vision to identify buildings in satellite imagery and combines this with population stock estimates. The post highlights the applications of these maps in disaster preparedness, climate adaptation, and sustainable energy, and provides examples of their use by organizations like the World Bank and World Vision. It also discusses the importance of accurate population data in low- and middle-income countries and the ongoing collaboration with external partners.

8/23/2024

This post details advancements in PyTorch for AI training and inference, focusing on large language models (LLMs). It covers memory-efficient fine-tuning techniques, architectural and algorithmic approaches for consumer-grade hardware, and challenges and solutions for on-device LLM deployment, including quantization.

8/22/2024

This post details the co-design process behind the second generation of Meta's MTIA (Meta Training and Inference Accelerator). It covers the integration with the PyTorch software ecosystem and model architectures for Meta's key applications, highlighting how MTIA achieves performance, efficiency, and developer experience for production model launches. Specific co-design examples are provided where silicon features accelerate Meta's models.

8/21/2024

This post details the engineering efforts behind bringing Llama 3 to life, covering data preparation, large-scale training, and inference optimization. It highlights the use of diverse, voluminous, and fresh data for GenAI, the infrastructure investments for training LLMs at scale, and the parallelism techniques employed to scale LLM inference for product applications.

8/20/2024

This post highlights Meta's ongoing efforts in AI infrastructure development, specifically focusing on the challenges and solutions for scaling AI technologies. It mentions the development of custom silicon (MTIA), large-scale GPU clusters, and open hardware designs for data centers. The post also references a keynote presentation discussing the AI landscape, technical challenges, and the role of open models and hardware in advancing AI.

8/14/2024

This post details the technical optimizations implemented for Meta AI's image animation feature to achieve large-scale, low-latency generation. Key contributions include: halving floating-point precision (float32 to float16/bfloat16) for reduced memory footprint and faster operations; improving temporal-attention expansion by replicating context tensors after cross-attention layers to reduce compute and memory; leveraging DPM-Solver to decrease sampling steps to 15; combining guidance and step distillation to reduce three forward passes per step to one, requiring only eight solver steps with one U-Net pass per step; and implementing PyTorch optimizations like TorchScripting and freezing for graph simplification and constant transformation, later migrating to PyTorch 2.0 for granular component optimization and advanced parallelism. For deployment, a traffic management system was developed to route requests regionally, minimizing network overhead and latency, and a retry-based probing system was implemented to approximate queueing and prevent failures when near capacity.

7/16/2024

Introduced AI Lab, a pre-production framework for continuously A/B testing common ML workflows to minimize Time to First Batch (TTFB) and prevent regressions. AI Lab uses an auto-shrinker to reduce compute resource consumption for tests and integrates with Incident Tracker for regression attribution. It was instrumental in achieving a 2x increase in TTFB improvements for the Python Cinder runtime and preventing an unrelated blocking regression during its rollout.

7/10/2024

This post details Meta's systematic framework for ensuring 'prediction robustness' in its large-scale machine learning recommendation models, particularly for the advertising business. It outlines the unique challenges of ML robustness, such as stochasticity and frequent model updates, and describes Meta's approach involving prevention guardrails, fundamental understanding, and intrinsic robustness. Specific solutions are highlighted for model, feature, training data, and calibration robustness, along with the use of ML interpretability tools like Hawkeye for debugging. The post also discusses how prediction robustness improves ML ranking performance and engineering productivity.

6/24/2024

This post details the development and implementation of an AI-assisted root cause analysis system for streamlining system reliability investigations. It introduces a novel approach combining heuristic-based retrieval and LLM-based ranking (using Llama models) to identify potential code changes responsible for incidents. The post highlights the training process involving continued pre-training and supervised fine-tuning of Llama 2 (7B) models with internal data and a dedicated RCA SFT dataset, achieving 42% accuracy in identifying root causes at the creation time of investigations. It also discusses future directions for AI in incident response, emphasizing explainability, confidence measurement, and proactive incident detection.

6/13/2024

This post details Meta's engineering efforts in scaling the training of large language models (LLMs). It outlines the shift from training many smaller models to fewer, extremely large ones, and the resulting infrastructure challenges. Key technical contributions include: addressing hardware reliability through rigorous testing and automation; implementing fast recovery mechanisms by reducing re-scheduling overhead and enabling rapid training re-initialization; developing efficient checkpointing and state preservation for training data; and optimizing GPU connectivity through robust, high-speed network infrastructure. The post elaborates on innovations across the infrastructure stack: training software (PyTorch integration), scheduling algorithms, hardware adaptations (e.g., modifying the Grand Teton platform, increasing GPU TDP, using HBM3, air-cooled environments), data center deployment strategies for optimal resource utilization, reliability measures for detecting and remediating failures (GPUs falling off, DRAM/SRAM UCE, HW network cables), and network infrastructure advancements with dual RoCE and InfiniBand 24k GPU clusters. Specific network optimizations include assigning communication patterns to network topology layers, implementing network-aware collective communication algorithms, and enhancing network load balancing and routing. Finally, it highlights the need for efficient data storage solutions for LLM training datasets.

6/12/2024

This post details Meta's strategies for maintaining large-scale AI capacity, specifically focusing on GPU training clusters. It highlights the challenges of ensuring capacity guarantees, handling 'bad hosts,' minimizing interruption rates, ensuring rollout safety, and maintaining host consistency in a dynamic environment. The post introduces 'maintenance trains' for cyclic server maintenance and 'gradual rollouts' for software and firmware updates, emphasizing the need for careful testing and vendor collaboration. It also describes the role of OpsPlanner in orchestrating disruptive work and ensuring host consistency, along with safety features like autostop and automatic offboarding of failing upgrades.

3/12/2024

This post details the architecture and implementation of Meta's new 24k GPU AI clusters, designed for GenAI workloads like Llama 3 training. It covers hardware (Grand Teton, OpenRack), networking (RoCE with Arista 7800, InfiniBand with NVIDIA Quantum2), storage (Tectonic FUSE, Hammerspace NFS), and performance optimizations (network topology awareness, NCCL tuning, FP8 support, checkpointing). It also highlights efforts in debuggability (desync debug) and PyTorch evolution for large-scale training.

1/29/2024

This post details significant improvements to the ML/AI developer workflow by addressing slow build times and inefficiencies in packaging and distributing execution files. It introduces solutions like leveraging Buck2 with remote execution to mitigate build non-determinism and implementing a Content Addressable Filesystem (CAF) with Btrfs for incremental packaging and distribution of Python executables. These changes have led to double-digit percentage reductions in overhead for ML engineers.

1/18/2024

This post details the adoption of Lazy Imports and the Python Cinder runtime to accelerate machine learning development at Meta. It describes the 'time to first batch' (TTFB) challenge in ML training, explaining how Lazy Imports defer module loading until needed, reducing startup times. The post highlights compatibility challenges with existing libraries like PyTorch and Numba, and the need to balance performance with dependability by disabling Lazy Imports before the first batch. It also mentions the learning curve for developers and future plans for streamlining onboarding, enhancing tooling, and community collaboration.

12/19/2023

This post introduces HawkEye, a new toolkit developed as part of Meta's Prediction Robustness program. HawkEye provides end-to-end monitoring, observability, and debuggability for ML workflows, significantly reducing the time spent debugging production issues. It enables isolation of top-line product issues to specific model snapshots, prediction anomalies to features, and upstream causes of feature issues. It also aids in diagnosing model snapshots and training data issues, thereby enhancing the robustness and reliability of ML-based products.

10/18/2023

This post details the creation of Meta's first-generation AI inference accelerator, MTIA v1, and the ongoing development of custom AI silicon. It highlights the challenges of designing hardware for rapidly evolving AI software, the importance of in-house collaboration between silicon and software teams, and the strategic advantage of optimizing silicon for Meta's specific AI workloads. The post also touches on the development of the entire design and verification flow from scratch for MTIA v1 and the future focus on addressing bottlenecks in memory and inter-chip communication for next-generation chips.

9/7/2023

This post introduces Chakra execution traces, a new open, graph-based representation of AI/ML workload execution designed to address the limitations of traditional AI benchmarking. It details how Chakra traces capture compute, memory, and communication operations, their dependencies, timing, and resource constraints. The post highlights Meta's use of Chakra for benchmarking (via tools like Mystique and PARAM), visualization, and performance optimization, and outlines future plans for enhancing benchmarking capabilities and using AI to generate representative traces. It also emphasizes the importance of industry collaboration through MLCommons for broad adoption.

9/7/2023

Introduced Arcadia, a unified end-to-end AI system performance simulator designed to model the compute, memory, and network performance of AI training clusters. Arcadia takes various inputs including AI system plans, network topology, workload distributions, and hardware specifications to provide insights into AI model and workload performance, enabling data-driven design decisions for AI clusters and infrastructure. It addresses challenges in optimizing large-scale, multi-layered AI systems by providing a common source of truth for performance analysis and joint optimization across application, network, and hardware levels.

8/9/2023

This post details the evolution of Instagram's Explore recommendation system, highlighting the application of advanced machine learning models like Two Towers neural networks and a multi-stage ranking approach (retrieval, first-stage ranking, second-stage ranking, final reranking). It emphasizes the use of caching and pre-computation for scalability and flexibility, and the adaptation of ML techniques for personalized content discovery at a massive scale.

10/31/2022

This post details the application of causal inference and machine learning (ML) to optimize Instagram's daily digest push notifications. It moves beyond traditional click-through rate (CTR) models by framing the problem as a budget allocation challenge to maximize the incremental value of sending notifications. The approach involves designing a randomized experiment to collect data for training an uplift model, which estimates the incremental impact of sending a notification. To handle online scoring and stabilize sending rates, an order-preserving score transformation using an online quantile computation service is employed. This method successfully reduced notification sending volume while maintaining user engagement and improving user experience.

9/19/2022

This post details the evolution of Meta's data ingestion infrastructure for machine learning training. It highlights the growing demands driven by increased model complexity, data volume, and training throughput. Key contributions include the development of a disaggregated Data PreProcessing tier (DPP) to scale data ingestion and training hardware independently, the introduction of 'feature flattening' for more efficient selective feature reading (yielding 2-2.3x data reading efficiency), and optimizations for memory consumption and bandwidth in the data reading tier, including client-side rebatching and the 'FlatMaps' project. The post also emphasizes the impact of data center power constraints on training accelerator deployment and the need to optimize ingestion to avoid becoming an ingestion-bound bottleneck.

6/14/2022

This post details the application of federated learning with differential privacy (FL-DP) to protect user data on mobile devices. It introduces a new system architecture and methodology designed to address challenges specific to FL-DP, such as label balancing, feature normalization, and metrics calculation without data visibility, slower mobile release cycles, and slower training due to device-side federation. The architecture combines infrastructure across mobile devices, trusted execution environments, and conventional back-end servers, and has been validated with an in-house FL library compatible with Meta's apps, showing minimal performance degradation compared to server-trained models.

8/4/2021

Open-sourced Winterfell, a general-purpose STARK prover and verifier implemented in Rust. This library aims to simplify the use of zero-knowledge proofs (ZKPs) for developers by abstracting away much of the cryptographic complexity. Winterfell leverages the properties of STARKs, such as minimal cryptographic assumptions, transparency, and performance, making it suitable for applications in blockchain privacy and scalability, as well as general computational integrity proofs. The post details the technical advantages of STARKs, the design of Winterfell, its modular crate structure, and provides guidance on using it through tutorials and examples.

7/15/2021

This post introduces Fully Sharded Data Parallel (FSDP), a new tool developed by FAIR Engineering to address the engineering complexity and high computational cost of training large AI models. FSDP shards model parameters across data parallel workers, enabling more efficient training of significantly larger models with fewer GPUs. It offers a simpler conceptual model compared to other parallelism techniques and improves memory and computational efficiency through communication and computation overlapping. The post details how FSDP works, its benefits over standard data parallel training, and provides guidance on its usage in language models, computer vision models, and through PyTorch Lightning and the FairScale library.

6/28/2021

Introduced Asicmon, a scalable observability framework for AI accelerators that abstracts custom interfaces and provides a standard interface to internal tools, facilitating load balancing, performance monitoring, and automated health checks. Developed Asimov, a custom specification language to simplify the development and rapid prototyping of new accelerators by generating efficient C++ monitoring modules. Introduced Atrace, an accelerator tracing solution for remote trace collection and detailed inspection of accelerator systems.

7/30/2019

Introduced EGG, a toolkit for multi-agent language emergence simulations. EGG allows researchers and developers to create game simulations where two neural network agents devise their own discrete communication system to solve a task. It lowers the barrier to entry for studying language emergence by providing a framework for designing and testing new games, and enabling exploration of questions related to the origins and properties of communication systems.

5/21/2019

This post introduces Pythia, a new deep learning framework built on PyTorch for vision and language multimodal AI models. It supports multitasking, distributed training, custom losses, metrics, scheduling, and optimizers, and includes reference implementations for winning AI competition entries. The goal is to accelerate research by improving reproducibility and enabling faster prototyping.

5/1/2019

This post details the use of PyTorch for accelerating AI research to production deployment in computer vision, NLP, and machine translation. It also covers the use of Infer, Sapienz, and idb for reliable code at scale, ML for developer productivity (code search, recommendation, bug fixing), building lightweight Android apps, and adaptive experimentation for product optimization.

4/4/2019

Introduced an experimental back end for the Glow compiler to target Habana's hardware accelerators. This is the first vendor-specific accelerator back end for Glow, enabling partners to design and optimize new silicon products for ML and strengthening the ecosystem for accelerating neural network ML workloads at production scale. The back end targets Habana's inference accelerator cards and is intended to improve ML performance on supported hardware, with applications in computer vision, recommendation, personalization, and natural language processing.

4/3/2019

Introduced PyTorch-BigGraph (PBG), a new tool for training multi-relation graph embeddings on extremely large graphs (billions of nodes, trillions of edges). PBG shards nodes and edges, performs multi-threaded training (single or multiple machines), and outputs node embeddings. It is built on PyTorch, allowing for custom loss functions and models. PBG is designed to be orders of magnitude faster than commonly used embedding software while producing comparable quality embeddings, making graph embedding accessible to researchers and organizations with large graph datasets but limited specialized computing resources.

3/27/2019

This post highlights the Turing Award presented to Yann LeCun, Geoffrey Hinton, and Yoshua Bengio for their breakthroughs in deep neural networks. It specifically mentions Yann LeCun's work at Facebook, including the back-propagation algorithm developed with Leon Bottou, which serves as a foundational principle for deep learning software platforms like PyTorch and TensorFlow.

3/14/2019

This post details the development and deployment of application-specific hardware (ASICs) for accelerating Facebook's infrastructure, specifically for AI training (Zion), AI inference (Kings Canyon), and video transcoding (Mount Shasta). It highlights the co-design efforts with partners, the adoption of the Open Compute Project (OCP) accelerator module (OAM) standard, and the architectural details of each hardware platform. The post also introduces the Glow compiler as a means to abstract hardware specifics for AI workloads.

3/12/2019

This post summarizes several AI research advancements, including the development of the Smart Camera for Portal, a new technique for safety-critical neural network verification, discussions on the future of deep learning hardware, teaching AI physics concepts, and a Lyapunov-based approach for safe reinforcement learning.

2/15/2019

FAIR open-sourced a retrained model of ELF OpenGo using reinforcement learning, which outperforms previous versions. This release includes 20 million self-play games, 1,500 intermediate models, a Windows executable bot for training aid, a new data archive, and an analysis of 87,000 professional Go games over 300 years. An interactive tool provides visualization of this archive and analysis. A new paper details insights that could help researchers understand the underlying mechanisms and apply them beyond Go.

2/5/2019

Introduced a new cross-lingual pretraining approach for NLP tasks, including a translation language modeling (TLM) objective that extends masked language modeling (MLM) by concatenating parallel sentences. This method achieved state-of-the-art results in unsupervised and supervised machine translation, and cross-lingual text classification for low-resource languages. Released code and pretrained models to the community.

1/31/2019

This post details the ZeroSpeech 2019 challenge, a collaboration to teach AI systems to learn speech using only audio input, without text or phonetic labels. Participants are tasked with building an AI system that can discover its own discrete 'orthographic' notation and use it to re-synthesize sentences. The challenge aims to advance unsupervised learning techniques and benefit AI tasks for low-resource languages.

1/30/2019

This post introduces a novel baseline for sentence embeddings by exploring various methods for computing sentence representations from pretrained word embeddings without any additional training. It investigates the effectiveness of random parameterizations over pretrained word embeddings, including bag of random embedding projections, random LSTMs, and echo state networks, demonstrating that these can sometimes match the performance of well-known sentence encoders like SkipThought and InferSent. The post also provides recommendations for proper experimental protocol for sentence classification evaluation.

1/22/2019

This post details the expansion and open-sourcing of the LASER (Language-Agnostic SEntence Representations) toolkit, enabling zero-shot transfer of NLP models across over 90 languages. The core technical contribution is the development of a single, shared encoder and decoder architecture (based on bidirectional LSTMs) that embeds all languages into a unified semantic vector space. This approach allows for language-independent representations and significantly improves performance on cross-lingual tasks like natural language inference and document classification, especially for low-resource languages. The post also highlights performance metrics, including processing speed and state-of-the-art results on various benchmarks, and discusses the underlying training methodology using joint byte-pair encoding and parallel corpora.

1/8/2019

This post details advancements in AI learning through semi-supervised and unsupervised training, including new methods for training NMT models on unsupervised data and expanding automatic translation services to more languages. It also highlights the use of user-supplied hashtags for image recognition training and the development of omni-supervised learning through data distillation. The post further elaborates on accelerating the transition from AI research to production with the release of PyTorch 1.0, which integrates Caffe2 and ONNX for a streamlined AI development pipeline. New tools and platforms extending PyTorch's capabilities, such as QNNPACK, FBGEMM, PyText, and Horizon (an RL platform), are introduced. Additionally, Glow, a framework for hardware acceleration of ML, and the Big Basin v2 ML-optimized server design are discussed. The transition of Oculus Research to Facebook Reality Labs and new explorations in AI and AR/VR, including the DeepFocus project, are also mentioned.

12/21/2018

This post introduces a new fully convolutional approach to automatic speech recognition and open-sources wav2letter++, the fastest state-of-the-art end-to-end speech recognition system. It also introduces flashlight, a fast, flexible standalone machine learning library with just-in-time compilation for CPU and GPU backends. The wav2letter++ toolkit is built on flashlight. This enables reproducible research and fast iteration for speech recognition tasks.

12/20/2018

Introduced and open-sourced Nevergrad, a Python library for derivative-free optimization. Nevergrad provides a standard ask-and-tell framework with a collection of gradient-free optimization algorithms (e.g., Differential evolution, Sequential quadratic programming, FastGA, Covariance matrix adaptation, Particle swarm optimization). It enables easier comparison of state-of-the-art methods for various machine learning problems, including multimodal, ill-conditioned, separable, discrete, continuous, mixed, and noisy problems. The library also includes tools for implementing and benchmarking algorithms, with specific examples for noisy and discrete optimization scenarios.

12/14/2018

Open-sourced PyText, a PyTorch-based NLP modeling framework designed to bridge the gap between experimentation and large-scale deployment. PyText offers a simplified workflow, prebuilt model architectures, and utilities for text processing and vocabulary management. It enables faster experimentation and deployment of NLP systems for tasks like document classification, sequence tagging, and semantic parsing. PyText has been used internally to reduce NLP model implementation time from weeks/months to days and operates at production scale, handling over a billion daily predictions with stringent latency requirements. It addresses the historical trade-off between research-oriented and production-optimized frameworks by leveraging PyTorch 1.0 and exporting models to Caffe2 for efficient, multithreaded C++ inference.

12/12/2018

Facebook contributes to MLPerf by open-sourcing Mask R-CNN2Go, a computer vision model optimized for embedded and mobile devices. This model supports object detection, classification, person segmentation, and body pose estimation, enabling real-time inference. Mask R-CNN2Go is based on the Mask R-CNN framework and is designed for mobile optimization, currently running on Caffe2 with future PyTorch 1.0 support. It is utilized in mobile ML use cases like AR effects and contributes to MLPerf's edge inference benchmarks.

12/7/2018

This post announces the stable release of PyTorch 1.0, highlighting its new features such as a hybrid front end for seamless execution mode transitions, revamped distributed training, a pure C++ front end, and deep cloud integration. It emphasizes the acceleration of the AI workflow from research to production and the growth of the PyTorch community, evidenced by its rapid expansion on GitHub and the success of educational programs and community-contributed projects like Horovod, PyTorch Geometry, and TensorBoardX. It also details enhanced cloud platform support from AWS, Google Cloud Platform, and Microsoft Azure.

12/5/2018

This post details FAIR's five-year anniversary and highlights key technical accomplishments. It introduces memory networks for improved long-term memory in neural networks, discusses the advancement and application of self-supervised learning (SSL) and generative adversarial networks (GANs) including the development of Wasserstein GAN (WGAN) for more stable training, and presents fastText for rapid and scalable text classification that handles subword units and supports numerous languages. The post also mentions contributions to cutting-edge translation research.

12/1/2018

Introduced a framework for using convolutional neural networks (CNNs) on satellite imagery to identify disaster-affected areas. This method trains models on general road and building datasets, then computes relative changes between pre- and post-disaster imagery to pinpoint areas of maximum change. A new metric, the Disaster Impact Index (DII), was proposed to quantify detected changes.

11/28/2018

Introduced Neural Egg Separation (NES), a semi-supervised ML method for isolating audio and visual sources from noisy/cluttered environments. NES iteratively mixes known and unknown signal distributions, allowing the model to improve at extracting the known signal. Experiments show NES outperforms similar supervision levels and is competitive with full supervision.

11/26/2018

This post details the release of new open source AI research tools and data as part of the fastMRI project, a collaboration between Facebook AI Research (FAIR) and NYU School of Medicine. It introduces new AI models and baselines for accelerating MRI scans, along with the first large-scale MRI dataset of its kind. The contribution includes open source tools for single-coil and multi-coil reconstruction, baseline models based on U-nets, and a leaderboard for comparing research results. The goal is to spur development of AI systems to speed MRI scans by up to 10x and improve diagnostic imaging technology.

11/19/2018

This post details research into AI agents' language learning capabilities, specifically their ability to communicate about images. It highlights a finding that agents may rely on low-level visual feature similarities rather than conceptual understanding, and discusses the implications for designing future AI communication systems.

11/8/2018

This post introduces a novel approach to optimizing floating-point arithmetic for AI hardware, aiming to improve efficiency and accuracy. The technique involves reducing word size, rethinking fixed-point and floating-point machinery, and adapting quantizers to data distributions, particularly for neural networks. This contrasts with traditional integer quantization methods and offers benefits such as faster R&D, easier deployment, and avoidance of overfitting issues associated with integer quantization. The research paper and hardware designs are publicly available.

11/7/2018

Open-sourced FBGEMM, a high-performance kernel library for server-side inference, optimized for low-precision calculations. Achieved >2x performance gains in production by enabling efficient quantized inference on CPUs. FBGEMM supports native tensor formats and fuses operations to minimize overhead, contributing to significant speedups and reduced bandwidth usage in various AI services like translation, recommendation systems, and text understanding.

11/7/2018

This post introduces and open-sources QNNPACK and FBGEMM, two high-performance kernel libraries designed to accelerate AI model inference on mobile devices and servers. QNNPACK focuses on optimizing computer vision models for mobile, while FBGEMM speeds up server-side operations for language translation, recommendation systems, and text understanding. Both libraries leverage low-precision integer arithmetic for inference, reducing computational overhead without significant accuracy loss, and are deployed in production at Facebook.

11/5/2018

Introduced a new, more accurate zero-shot learning (ZSL) model called GAZSL, which uses generative adversarial networks (GANs) to analyze text articles and visually identify objects described within them. This model extracts key visual features from text, creates synthetic visualizations, and uses these to classify objects, outperforming existing ZSL algorithms by 4-7% on average across benchmarks. The model was open-sourced.

10/29/2018

Introduced QNNPACK, a high-performance kernel library optimized for mobile AI, which speeds up operations like depthwise convolutions used in advanced neural network architectures. QNNPACK has been integrated into Facebook apps and deployed to billions of devices, outperforming state-of-the-art implementations by approximately 2x on benchmarks like quantized MobileNetV2. The library provides high-performance implementations of convolutional, deconvolutional, and fully connected operations on quantized tensors, addressing the need for efficient inference on mobile devices with limited computational power. It leverages a linear quantization scheme compatible with the Android Neural Networks API and is optimized for cases where panels of input matrices fit into L1 cache, fusing requantization and eliminating matrix repacking overhead.

10/26/2018

Introduced the XLNI dataset, which expands the MultiNLI corpus with 14 additional languages (including Swahili and Urdu) for evaluating cross-lingual natural language understanding (NLU) systems. This dataset comprises 112,500 annotated sentence pairs and includes baselines to aid in the creation of multilingual NLU systems, supporting research into training models in one language and applying them to others.

10/22/2018

Introduced the Symbol-to-Instrument Neural Generator (SING), a neural audio synthesizer that generates musical notes significantly faster than traditional AI systems. SING achieves this by producing audio in larger batches (up to 1,024 samples at once) and demonstrates a generation time 2,500x faster and training time 32x faster than state-of-the-art encoder-based systems like WaveNet, while maintaining high realism.

10/15/2018

Introduced dynamic meta-embeddings for NLP, a novel approach that dynamically selects the most useful word embeddings for a given task by training a neural network on multiple embeddings. This method outperforms traditional approaches using a single type of word embedding and provides insights into how NLP systems prioritize embeddings.

10/3/2018

This post details the application of AI within Facebook Marketplace, highlighting its role in enhancing buyer and seller experiences. It showcases the use of computer vision and NLP for product indexing and retrieval, visual similarity searches, machine translation, and seller tools like category and pricing autosuggestions. The post also elaborates on the technical implementation, including multimodal ranking systems (Lumos, DeepText), the use of FAISS for similarity search, and distributed training strategies to improve model performance and scalability. Additionally, it discusses the integration of AI into Messenger through the M assistant for enriched buyer-seller interactions.

10/2/2018

This post announces the release of PyTorch 1.0, which unifies PyTorch's research flexibility with Caffe2's production capabilities. Key technical advancements include a new hybrid front end for tracing and scripting models, a revamped torch.distributed library for faster distributed training, and an eager mode C++ interface for performance-critical research. The post also details expanded ecosystem support from cloud providers (AWS, Google, Microsoft) and hardware partners (ARM, Intel, IBM, NVIDIA, Qualcomm) to accelerate AI development and deployment.

10/1/2018

Introduced composable planning as a new approach to building AI agents capable of solving unfamiliar tasks. This method trains agents on a set of simple, related tasks within a given environment to enable them to perform longer, more complex tasks by learning an environment model based on assigned attributes and then using this model to plan novel tasks. This research aims to create more versatile AI agents beyond single-purpose systems.

9/24/2018

Introduced data distillation, a method for omni-supervised learning that combines supervised and unsupervised data. The process involves training a model on supervised data, applying it to unsupervised data to generate labels, and then retraining the model with both datasets. This approach demonstrated the potential to surpass state-of-the-art fully supervised methods.

9/13/2018

This post introduces Glow, an open-source compiler framework developed by Meta to enable hardware acceleration for Machine Learning (ML) workloads. Glow accepts computation graphs from deep learning frameworks like PyTorch and generates optimized code for a variety of ML accelerators. It features hardware-independent optimizations, a configurable memory allocator, and an instruction scheduler. The post highlights industry collaboration with partners like Cadence, Esperanto, Intel, Marvell, and Qualcomm to support Glow in future silicon products, aiming to accelerate inference performance and reduce development time for hardware partners.

9/11/2018

Introduced Rosetta, a large-scale ML system for extracting text from images and videos. Developed a two-step text extraction pipeline: text detection using a modified Faster R-CNN with ShuffleNet, and text recognition using a ResNet18 CNN with CTC loss and LSTMs. Implemented curriculum learning and learning rate scheduling for CTC training. Addressed multilingual support, including right-to-left languages, with a unified model. Enhanced training data generation with a modified SynthText engine for Unicode compatibility and language-specific rules. Optimized inference for real-time performance on CPU machines with Caffe2.

9/7/2018

This post details significant optimizations to the neural machine translation (NMT) training pipeline, reducing training time from weeks to 32 minutes. Key contributions include: switching to 16-bit precision with overflow guarding, implementing cumulative gradient updates to reduce communication overhead and increase effective batch size, and overlapping the backward pass with gradient synchronization across distributed workers. These techniques enable training on larger datasets, including monolingual data via back-translation, leading to more accurate NMT models. Inference speed was also improved through caching and batching by word count.

8/31/2018

Introduced a novel unsupervised machine translation approach that significantly improves accuracy (over 10 BLEU points) for low-resource languages by leveraging word embeddings, adversarial training for bilingual dictionary inference, and back translation with language models. This method allows translation without parallel text, using only monolingual data.

8/20/2018

This post details a research collaboration between Facebook AI Research (FAIR) and NYU School of Medicine to improve MRI scan speed using AI. The project, fastMRI, aims to make MRI scans up to 10 times faster by training neural networks to reconstruct high-quality images from less raw data. This involves challenges in bridging gaps in scanning data without sacrificing accuracy, and the project will open-source AI models, baselines, and evaluation metrics, while NYU will open-source the image dataset.

7/13/2018

This post highlights the "Test of Time" award received by Ronan Collobert and Jason Weston for their 2008 ICML paper, "A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning." The paper introduced a novel approach to NLP by integrating multiple tasks into a single deep neural network trained jointly, avoiding cascading errors from pipelined systems. Key influential contributions included: training semantically meaningful word vectors via neural networks, using multitasking and auxiliary tasks for NLP, employing convolutional neural nets in NLP, and a form of attention mechanism. The post details the initial reception of the work, its eventual mainstream adoption, and its impact on current NLP research and products like Fairseq and Elmo. It also touches upon the researchers' current focus on deeper language understanding, dialogue systems, and speech recognition.

7/11/2018

Introduced the 'Talk the Walk' research task for embodied AI, using 360-degree images of New York City for navigation. Developed a novel attention mechanism called MASC (Masked Attention for Spatial Convolution) to improve the guide bot's ability to focus on the map and predict the tourist bot's location, achieving over twice the accuracy on the test set. Explored emergent communication protocols (continuous vectors and synthetic language) and compared their performance against natural language, finding synthetic communication to be more efficient but less human-understandable. Released baselines and the dataset for 'Talk the Walk' to the research community.

5/3/2018

Introduced Embodied Question Answering (EmbodiedQA), a new AI task requiring agents to combine perception, navigation, and communication to answer questions in virtual environments. Developed House3D, a collection of 45,000 simulated indoor environments for training and testing these agents. Proposed a modular navigation approach splitting tasks between a planner and controller. Explored learning capabilities including active perception, commonsense reasoning, language grounding, and credit assignment using reinforcement learning, imitation learning, and reward shaping. Released EmbodiedQA data and House3D as open source.

5/2/2018

This post announces the upcoming beta of PyTorch 1.0, which unifies research and production capabilities with a hybrid execution mode and ONNX integration. It also details the expansion of ONNX to support more frameworks, the release of ResNext3D for video understanding, the open-sourcing of the Translate library and MUSE for machine translation, and research advancements in computer vision (image recognition model, DensePose), reinforcement learning for autonomous agents (EmbodiedQA, House3D), and game AI (ELF OpenGo). The post also highlights the company's commitment to responsible AI development.

5/2/2018

Introduces PyTorch 1.0, a unified framework that merges PyTorch's research flexibility with Caffe2's production capabilities and ONNX's interoperability. This enables a seamless transition from research prototyping to production deployment by offering a hybrid front end that supports both imperative and declarative execution modes. The post details the technical challenges of migrating from PyTorch to Caffe2 for production and how ONNX addresses this. It also announces the open-sourcing of related AI tools like Translate, ELF, Glow, Tensor Comprehensions, and Detectron to support end-to-end deep learning system development.

5/2/2018

This post details a novel approach to advancing image recognition by training models on large-scale public image datasets labeled with user-supplied hashtags. It addresses the limitations of manual labeling for massive datasets by using hashtags as weak supervision. The post highlights the development of new approaches to handle multiple labels per image, hashtag synonyms, and balancing frequent/rare hashtags. It also describes the engineering effort involved in distributed training across 336 GPUs to shorten training time from over a year to a few weeks, and the development of duplicate removal methods. The research achieved record-high accuracy on ImageNet and demonstrated significant performance boosts on COCO object-detection, showcasing the effectiveness of weakly supervised pretraining with hashtags.

5/2/2018

This post details the expansion of the Open Neural Network Exchange (ONNX) format to improve interoperability between AI frameworks and hardware runtimes. It highlights new support for Baidu's PaddlePaddle, Qualcomm SNPE, and Apple's Core ML, enabling AI engineers to move models more easily between development frameworks and production serving environments. The post also discusses the ongoing efforts to expand ONNX support for NLP models and create a common framework interface for optimized execution on various hardware.

3/20/2018

Introduced Big Basin v2, an upgraded AI hardware system featuring NVIDIA Tesla V100 GPU accelerators, an upgraded head-node (Tioga Pass) for increased CPU performance, doubled PCIe bandwidth between CPUs and GPUs, and an upgraded OCP network card for enhanced network bandwidth for distributed training. This resulted in a 66% increase in single-GPU performance and near-linear performance improvement in large-scale distributed GPU training. The post also details how Big Basin v2 integrates with the FBLearner platform (Feature Store, Flow, Predictor) and other custom hardware like Bryce Canyon and Twin Lakes.

2/21/2018

This post details the technical evolution of Meta's AI-powered suicide prevention tools. It describes the transition from relying on user reports to using machine learning for proactive detection. Key technical details include: the use of FBLearner for training classifiers, the application of n-gram-based linear regression for initial text classification, the incorporation of DeepText neural networks for more nuanced text understanding, and the use of a random forest algorithm to combine various signals (post text, comments, time of day, day of week, post type, reaction counts) into a final risk score. It also highlights the development of multilingual classifiers to extend coverage to more languages.

1/24/2018

This post introduces the development and application of multilingual embeddings as a novel technique to scale Natural Language Processing (NLP) across Facebook's diverse user base speaking over 100 languages. It details the challenges of language-specific NLP, the drawbacks of translation-based approaches, and the methodology for training and integrating multilingual embeddings into the DeepText framework. The post highlights the benefits of this approach, including faster feature deployment to new languages, improved accuracy, and significant latency reduction, and outlines ongoing research directions.

12/8/2017

This post announces the production-ready release of ONNX V1, an open format for deep learning frameworks. It highlights the collaborative effort with AWS and Microsoft, community support from various hardware and software vendors, and integration with frameworks like Caffe2, Microsoft Cognitive Toolkit, Apache MXNet, PyTorch, and TensorRT. The post also mentions community-contributed converters for TensorFlow and outlines future plans to expand ONNX support beyond vision applications to areas like language modeling.

9/21/2017

This post details the development and deployment of Simultaneous Localization and Mapping (SLAM) technology for mobile devices, enabling immersive AR experiences. It highlights the engineering challenges in optimizing SLAM algorithms for diverse mobile devices, reducing binary size, and creating believable AR experiences through the WorldTracker API. The post also showcases the first AR-driven art project with artist Heather Day, demonstrating the potential of AR to blend art and technology.

8/3/2017

This post details the transition from phrase-based machine translation to neural machine translation (NMT) for all of Facebook's backend translation systems. It introduces the use of sequence-to-sequence LSTMs with attention for improved context handling and accuracy, addressing challenges with word order and unknown words. The post also discusses vocabulary reduction techniques for efficiency and the use of FBLearner Flow for hyperparameter tuning. It highlights the scaling of NMT with Caffe2, including memory optimizations and specialized libraries for inference. Finally, it mentions the integration of CNN models for machine translation, achieving significant BLEU score improvements, and outlines future work on multilingual models and incorporating richer context.

7/26/2017

This post details the application of AI, specifically style transfer, to create visual effects for a VR film ('Alteration'). It highlights the challenges of adapting style transfer for high-definition, stereoscopic VR content, including technical hurdles like memory constraints and the subjective nature of aesthetic evaluation. The post describes retraining models with larger images and the computational effort involved in applying these effects to video frames, demonstrating AI's potential as a creative tool for filmmakers and opening new avenues for visual storytelling in VR.

7/6/2017

Introduced ELF (Extensive, Lightweight, and Flexible) platform for game research, enabling AI model training in diverse game environments (board games, Atari, custom RTS). ELF supports high-performance computing with minimal resources (laptop with GPU), handles concurrency automatically, and offers a clean Python interface for batch processing of game states. It includes Mini-RTS, a fast RTS engine for research, and has shown faster simulation and training speeds than OpenAI Gym. The platform's architecture involves a C++ simulator communicating with Python AI models, with flexible pairing of game instances and actor models for rapid algorithm prototyping.

6/14/2017

Introduced dialog agents with the capability to negotiate, utilizing multi-issue bargaining tasks. Key technical innovation is 'dialog rollouts,' a technique where an agent simulates future conversations to choose utterances with maximum expected future reward. Agents were trained end-to-end on human negotiation data and fine-tuned using reinforcement learning, achieving performance comparable to human negotiators and demonstrating novel negotiation tactics and sentence generation.

6/8/2017

This post details a significant acceleration in machine learning training for computer vision. It describes the use of Caffe2, Gloo library for collective communication, and Big Basin GPU servers to reduce ImageNet-1k training time from days to one hour with leading accuracy. This represents a substantial infrastructure and algorithmic improvement for large-scale deep learning.

5/15/2017

Introduced ParlAI, a new open-source software platform for dialog research. ParlAI unifies dialog tasks and agents into a single, shared repository, facilitating the training and testing of dialog models. It includes over 20 public datasets across five categories: question answering, sentence completion, goal-oriented dialog, chit-chat dialog, and visual dialog. The platform features a standardized observation/action object format for agent communication and integrates with Mechanical Turk for data collection, training, and evaluation. Initial implementations include baseline agents and neural network models like end-to-end memory networks and attentive LSTM models.

5/9/2017

Introduced a novel convolutional neural network (CNN) approach for neural machine translation, achieving state-of-the-art accuracy at nine times the speed of recurrent neural systems. The approach utilizes multi-hop attention and gating mechanisms for improved translation quality and efficiency. The fairseq toolkit and trained systems were open-sourced.

5/2/2017

This post details the expansion of the fastText library to support smaller-memory devices by reducing model size to a few hundred kilobytes. This was achieved through a collaboration with the FAISS library and the implementation of compression techniques described in "FastText.zip: Compressing Text Classification Models." The post also highlights fastText's efficiency, speed, and ease of use for text classification and word representation learning, including its use of low-dimensional vectors, hierarchical softmax, and n-gram models. New pre-trained vectors in 294 languages and quick-start tutorials are also released.

3/29/2017

This post introduces Faiss, a new library for efficient similarity search on billion-scale datasets. It details the challenges of similarity search with high-dimensional vectors generated by AI models, compares Faiss to existing software packages, and outlines its advantages in speed, memory usage, and GPU implementation. The post also discusses evaluation metrics (speed, memory, accuracy) and presents performance benchmarks on billion-scale datasets using both CPU and GPU implementations, including the construction of k-nearest-neighbor graphs.

3/27/2017

This post details significant advancements in optimizing gradient-boosted decision trees (GBDTs) for large-scale, real-time machine learning applications at Facebook. It introduces specific C++ implementation improvements, including compiled trees with annotations, range evaluation, and optimizations for common and categorical features, which resulted in up to a 5x performance improvement. These optimizations enable more frequent model updates and the use of larger, more precise models for ranking user experiences like notifications, News Feed, and recommendations, directly impacting the efficiency and effectiveness of AI-driven personalization for billions of users.

3/8/2017

Introduced Big Basin, a next-generation AI GPU server, as the successor to Big Sur. Big Basin enables training of 30% larger machine learning models due to increased arithmetic throughput and memory (16 GB vs 12 GB). It features a modular, scalable design with disaggregated CPU and GPU compute, allowing independent scaling of components and integration with existing OCP infrastructure like the Tioga Pass server platform. The GPU tray is swappable for future upgrades. The system uses external PCIe cables for head node to GPU connection, functioning as a 'just a bunch of GPUs' (JBOG). It offers improved serviceability and thermal efficiency by positioning GPUs in front of cool air intake. Big Basin is equipped with eight NVIDIA Tesla P100 GPUs connected via NVLink in a hybrid cube mesh, offering improved performance per watt with single-precision floating-point arithmetic increasing from 7 to 10.6 teraflops and introducing half-precision for higher throughput. Tested with ResNet-50, it achieved nearly 100% throughput improvement over Big Sur. The design specifications are open-sourced via the Open Compute Project.

2/2/2017

This post details the development and scaling of FBLearner Flow, a platform for machine learning pipelines, and Lumos, a computer vision platform built on top of it. It highlights the use of deep learning for image and video understanding, enabling features like automatic alt text (AAT) with action recognition and descriptive photo search based on object recognition and image embeddings. The post also discusses the technical aspects of training models using labeled data, semantic clustering, and the use of quantization techniques for image embeddings.

12/1/2016

This post provides a foundational explanation of Artificial Intelligence (AI), detailing its core concepts such as neural networks, deep learning, and convolutional neural networks (ConvNets). It categorizes AI learning into reinforcement, supervised, and unsupervised learning, and discusses the scientific basis of AI, contrasting it with science fiction portrayals. The post also touches upon the societal impact of AI and offers career guidance for aspiring AI professionals.

11/8/2016

This post details the evolution of Meta's AI capabilities, highlighting the development of AI platforms like FBLearner Flow, AutoML, and Lumos to accelerate the deployment of AI research into production. It also introduces Caffe2Go, a deep learning platform for mobile devices enabling real-time AI operations, and discusses the application of AI in VR (speech recognition for avatars, voice commands) and connectivity projects (computer vision for deployment planning). The post also outlines ongoing research into contextual understanding and reasoning for AI systems.

11/8/2016

This post details the development and deployment of Caffe2Go, a lightweight deep learning framework for mobile devices, enabling real-time AI inference. It describes optimizations for model size and quality for style transfer, including reducing convolutional layers, adjusting spatial resolution, and using instance normalization. The post also highlights the modular design of Caffe2Go, its integration with platform-specific optimizations like NNPack and Metal, and its developer-friendly deployment process.

10/25/2016

Introduced adaptive softmax, a novel softmax function approximation tailored for GPUs that exploits word distribution to form clusters, minimizing computational complexity for training neural language models over large vocabularies. Open-sourced torch-rnnlib, a library for designing and testing recurrent models on GPUs, with seamless access to fast baselines through torch.cudnn bindings. Demonstrated achieving 12,500 words/sec on a single GPU with adaptive softmax and torch-rnnlib, enabling state-of-the-art model training with limited resources.

8/25/2016

Introduced DeepMask and SharpMask algorithms for precise object segmentation in images, and MultiPathNet for object classification. DeepMask uses a patch-based binary classification approach, while SharpMask refines masks using features from earlier network layers. MultiPathNet leverages multiple paths to exploit multi-scale information and context for improved classification accuracy. The code for DeepMask+SharpMask and MultiPathNet, along with research papers and demos, were open-sourced.

8/18/2016

FAIR open-sourced fastText, a library for efficient text classification and learning word vector representations. FastText employs hierarchical classifiers with Huffman coding for speed, and uses bag of n-grams and subword information to improve accuracy. It achieves state-of-the-art performance on standard problems like sentiment analysis and tag prediction, often matching deep learning models in accuracy but being significantly faster for training and evaluation. It also supports learning word vector representations, outperforming word2vec and other methods, especially for morphologically rich languages.

6/23/2016

Introduced and open-sourced Torchnet, a software toolkit for the Torch community to build rapid and reusable prototypes of deep learning systems. Torchnet provides boilerplate code, key abstractions, and reference implementations to speed up development, encourage modular programming, and facilitate asynchronous, parallel data loading and efficient multi-GPU computations. It offers abstractions for Datasets, Dataset Iterators, Engines, Meters, and Logs, and includes implementations for computer vision, natural language processing, and speech processing.

6/20/2016

This post details advancements in stabilizing Generative Adversarial Networks (GANs) for unsupervised learning. It introduces Laplacian Adversarial Networks (LAPGAN) and Deep Convolutional Generative Adversarial Networks (DCGAN) as methods to improve GAN stability and enable visualization of learned features. The post also discusses the application of these techniques to video generation using Adversarial Gradient Difference Loss Predictors (AGDL), aiming to equip machines with the ability to predict future states of the world and develop common sense.

6/1/2016

Introduced DeepText, a deep learning-based text understanding engine utilizing convolutional and recurrent neural networks, word/character-level learning, and word embeddings. It achieves near-human accuracy in understanding textual content across multiple languages, reducing reliance on language-specific preprocessing. DeepText is trained using FbLearner Flow and Torch, and deployed via the FBLearner Predictor platform for scalable model distribution. It has been applied to intent detection in Messenger and for identifying content related to selling items, with future applications including sentiment analysis, entity extraction, and joint understanding of textual and visual content.

5/10/2016

This post introduces fundamental AI concepts including Machine Learning, Gradient Descent, Deep Learning, Back Propagation, and Convolutional Neural Networks, explaining their underlying principles and applications in areas like image recognition and text translation.

5/9/2016

Introduced FBLearner Flow, a new platform designed to democratize machine learning model training and experimentation for all Facebook engineers. FBLearner Flow allows for easy reuse of algorithms, parallelized training across machines, and simplified experiment management through a UI. Key concepts include Workflows, Operators, and Channels. The platform consists of an authorship and execution environment, an experimentation management UI, and predefined pipelines. It supports automatic UI generation from pipeline definitions and parallelization of Python code using futures. The system compiles workflows into a DAG for execution and manages resource allocation for operators. The UI provides a structured form for launching workflows based on typed input schemas and allows for visualization and comparison of experiment outputs.

4/13/2016

This post details the development and deployment of an AI backbone at Facebook, powered by a 40 PFLOPS GPU cluster, enabling over 25% of engineers to run 50x more AI experiments daily. It highlights the application of AI in automatic translation for 800 million users, photo image search based on content, image segmentation for pixel-level object understanding, and real-time video classification using spatio-temporal convolutions.

3/30/2016

This post details the engineering behind the 'On This Day' feature, highlighting the application of AI and machine learning to personalize user memories. It showcases the use of UX research, automatic filtering (including blocking exes and dismissed content), and a machine-learning ranking model. The ranking model leverages personalization signals (user interactions, demographics) and content understanding via a computer vision platform that recognizes objects, scenes, and objectionable content in photos and videos. This post represents a significant step in applying AI for personalized content surfacing and memory recall.

2/25/2016

This post announces the launch of the Facebook AI Research Partnership Program, which donates GPU servers to European research institutions to accelerate AI and machine learning research. It highlights the importance of collaboration and open-source contributions in advancing AI.

12/10/2015

Introduced "Big Sur," a new generation of Open Rack-compatible GPU-based hardware designed for large-scale AI training. Big Sur features eight high-performance GPUs, configurable PCI-e topologies, and is twice as fast as previous generations. It is optimized for thermal and power efficiency and designed for operational efficiency and serviceability with toolless component replacement. The design is being contributed to the Open Compute Project.

6/19/2015

This post highlights a significant achievement in computer vision research with the award-winning paper 'DynamicFusion: Reconstruction and Tracking of Non-rigid Scenes in Real-Time'. This work represents a breakthrough in real-time 3D scanning of moving scenes, directly contributing to advancements in virtual and augmented reality by enabling more realistic integration of the real world into these digital environments.

6/2/2015

This post details the challenges and innovations in scaling collaborative filtering (CF) for recommendation systems at Facebook's massive scale. It introduces the problem of handling billions of ratings and users, discusses the limitations of standard distributed approaches for matrix factorization (SGD and ALS), and presents a novel 'rotational hybrid approach' that significantly reduces network traffic and overcomes skewed item degree distributions by extending the Apache Giraph framework with worker-to-worker messaging. It also discusses the blend of SGD and ALS for improved performance and accuracy.

6/2/2015

This post announces the expansion of the Facebook AI Research (FAIR) team to Paris, with researchers focusing on image and speech recognition, and natural language processing. This contributes to the ongoing evolution of Meta's AI research and development efforts.

1/16/2015

This post details the open-sourcing of optimized deep-learning modules for Torch, including GPU-accelerated ConvNets with FFT-based convolutions and parallelization over multiple GPUs. It highlights engineering efforts to improve GPU convolution layers, achieving speedups of up to 23.5x compared to existing implementations. The release also includes optimized Lookup Tables, Hierarchical SoftMax, Cross-map pooling, a GPU implementation of 1-bit SGD, and a faster Temporal Convolution layer.

3/7/2012

This post describes Meta's research culture, emphasizing rapid iteration, risk-taking, and data-driven decision-making. It highlights the use of an experimentation framework (Gatekeeper for front-end, similar system for back-end) and access to anonymized interaction logs in Hadoop for objective metrics. The author shares personal experience of quickly prototyping and deploying a ranking idea, demonstrating the value of moving fast and failing early to iterate on product development.

12/15/2011

Introduced Kendo, a software-only prototype system providing a deterministic multithreading programming model. Kendo enforces deterministic interleaving of synchronization operations via a novel distributed scheduling algorithm that uses performance counters to construct deterministic logical clocks for load-balanced scheduling. It runs on commodity hardware with modest performance cost, simplifying debugging and testing of parallel applications by enabling repeatable results and systematic bug detection.