Internal Innovation and Feature Development Process

2/11/2026

This post introduces Just-in-Time Tests (JiTTests) as a novel approach to software testing, leveraging Large Language Models (LLMs) to automatically generate tests on the fly for specific code changes. This innovation aims to address the limitations of traditional testing in the face of rapid agentic development by eliminating manual test maintenance, reducing false positives, and focusing on catching serious bugs just-in-time before production.

1/16/2025

This post introduces the concept of Diff Authoring Time (DAT) as a metric for measuring developer productivity and discusses its application in experiments to assess the impact of features like type-safe mocking frameworks in Hack. It highlights the use of metrics to run experiments on productivity within Meta's internal codebase.

12/9/2024

This post details a collaborative survey conducted by Meta, JetBrains, and Microsoft on the state of Python typing in 2024. It highlights high adoption rates (88% always/often use types), key drivers (IDE tooling, documentation, bug catching), and persistent usability challenges (expressing complex patterns, tooling latency, lack of types in libraries, documentation discoverability). The survey also reveals diverse developer cohorts using typed Python and popular tooling configurations (VS Code with Mypy). Recommendations are provided for improving Python type system standardization, performance, and documentation accessibility.

10/17/2023

This post introduces Meta's Systematic Code and Asset Removal Framework (SCARF), a new internal tool designed to automate the complex process of product and feature deprecation. It details the challenges of safely removing code and data in a large organization, the failure modes that can arise without proper guidance, and how SCARF addresses these issues through workflow management, dependency analysis, and the creation of a deletion roadmap. The post highlights SCARF's ability to guide engineers, automate cleanup of unused assets, and work in conjunction with manual intervention to ensure efficient and safe deprecation.

11/16/2022

This post details Meta's efforts to improve code review time by introducing and refining several tools and processes. It introduces the 'Time In Review' metric (P75) as a key indicator of developer satisfaction and productivity, and 'Eyeball Time' as a guardrail metric. It describes the development and impact of 'Next Reviewable Diff,' a feature that uses ML to queue up relevant diffs for reviewers, leading to a 17% increase in review actions per day. It also explains improvements to the reviewer recommendation system, incorporating work hour awareness and file ownership, which resulted in a 1.5% increase in diffs reviewed within 24 hours and improved recommendation accuracy. Finally, it introduces 'Stale Diff Nudgebot,' which uses targeted pings to expedite reviews for slow-moving diffs, reducing average 'Time In Review' by 7% and the proportion of diffs waiting longer than three days by 12%.

7/18/2022

This post details the development of dynamic text animations and new text styles for Instagram Stories, highlighting the engineering challenges and solutions. It covers issues with font availability on Android and the development of a custom solution using Everstore and Haystack, the collaborative process between design and engineering to implement complex text styles like 'Directional', and the use of natural language processing for word-by-word reveal animations. It also discusses debugging native Android crashes related to font caching and the complexities of rendering various text features like ligatures, RTL text, and emojis on Android.

10/3/2019

Introduced Hydra, a lightweight Python framework for simplifying complex application development through composable and overridable configurations. Key features include dynamic command-line tab completion, remote job execution, and batch job launching, aimed at reducing boilerplate, speeding up development, and minimizing bugs in research projects and complex applications.

2/13/2019

This post highlights the recognition of engineering manager Mark Harman with the IEEE Harlan D. Mills Award for his fundamental contributions to software engineering, particularly in search-based software engineering and genetic improvement. It showcases how these research advancements have been directly applied at Meta through the development and use of Sapienz and SapFix, tools that automate software testing and bug fixing for Android and iOS apps. This demonstrates a concrete example of Meta's internal innovation process, where cutting-edge research is translated into practical, impactful engineering tools that enhance developer productivity and product quality.

6/28/2018

This post introduces Spiral, a system for self-tuning high-performance infrastructure services at Facebook scale using real-time machine learning. It addresses the challenge of rapidly changing codebases by replacing hand-tuned heuristics with machine learning models, allowing for optimization in minutes instead of weeks. Spiral enables engineers to define desired outcomes in code and provides a framework for continuous retraining and adaptation, exemplified by its application in automating reactive caching heuristics.

5/2/2018

This post introduces Sapienz, an intelligent automated software testing tool developed and deployed at Facebook. Sapienz leverages search-based automated test design to significantly accelerate the testing process for the Facebook Android app, enabling faster deployment of new features with improved stability and performance. It automates the design, execution, and reporting of tens of thousands of test cases daily, with a high rate of actionable reports, and integrates seamlessly into the continuous integration system.

12/14/2017

This post details advancements in Facebook's software development workflow in 2017, including the deployment of Jupiter for high-performance job matching, optimizations to the Buck build tool reducing compilation times, and the development of the One World system for managing tens of thousands of remote testing resources. It also highlights the expansion of Infer's capabilities for bug detection, including concurrency bugs, and its role in migrating Android News Feed to a multi-threaded model. The adoption of a continuous deployment model and the open-sourcing of React, Relay, Litho, Yarn 1.0, and various machine learning tools (Fairseq, ParlAI, ELF, ONNX) are also discussed.

9/6/2017

This post introduces the Infer static analyzer, highlighting its capability to find inter-procedural bugs at scale in large codebases (Java and C). It details how Infer's compositional, summary-based analysis, using pre/post specifications, allows it to identify bugs that intra-procedural analyzers miss, citing examples from DuckDuckGo and OpenSSL. This demonstrates a significant advancement in Meta's internal tooling for improving code quality and reliability, contributing to the overall feature development process.

8/31/2017

This post details the evolution of Meta's release engineering process from a branch/cherry-pick model with three daily pushes to a quasi-continuous "push from master" system for web, and the development of a comprehensive build and test stack for mobile. Key technical contributions include the transition to deploying directly from master for web, enabling tens to hundreds of diffs every few hours with tiered rollouts and the Gatekeeper system for risk mitigation. For mobile, it outlines a three-layered CI stack (builds, static analysis with Infer, and automated testing) that runs on every commit, leading to reduced release cycles from weeks to days. The post also highlights the elimination of hotfixes, better support for global engineering teams, and the development of next-generation tools and processes driven by the need for scale.

5/24/2017

This post introduces Jupiter, a high-performance job-matching service designed to address bottlenecks in the engineering workflow by efficiently distributing jobs to appropriate workers based on multidimensional capabilities. It highlights the importance of minimizing engineer waiting times for tasks like building packages, installing dependencies, and running tests, thereby increasing productive time. Jupiter's capability-matching approach, atomic job acquisition, and horizontal scalability are key contributions to optimizing the internal development and deployment process.

6/17/2016

This post details the development of Canvas, a new platform for creating immersive mobile ad experiences. It highlights how Canvas builds upon existing infrastructure and learnings from other teams (Videos, Paper, Instant Articles) and product initiatives (carousel format, Lead Ads). The post emphasizes the technical challenges of creating a flexible yet intuitive UI for designers, the design of a declarative language for configuring components and layouts, and the importance of real-time mobile previews. It showcases how Meta's internal innovation process allows for rapid development of complex features by leveraging existing technologies and observing user behavior to drive product evolution.

6/2/2016

This post details the engineering evolution of the Safety Check feature, highlighting the transition from manual processes to automated deployment and monitoring. It showcases the development of a robust activation algorithm leveraging the social graph, the implementation of rate-limiting services for stability, and the creation of proactive testing systems ('shadow launches'). Furthermore, it describes the automation of the launch process through an internal tool and the use of a Messenger bot for post-launch monitoring, demonstrating a significant advancement in the speed, reliability, and scalability of deploying critical features.

12/30/2015

This post highlights several engineering innovations that originated from hackathons and were developed into production features in 2015. These include: a prototype for real-time VR coding using React and three.js, an on-board optics design for data center efficiency, the Boomerang app for Instagram, the Stetho debugging platform for Android, the Warp Speed Data Transfer library for fast data migration, and the integration of OpenPGP for secure email communications. These examples showcase the impact of hackathons on product development and engineering infrastructure.

9/17/2015

This post details the development and implementation of homegrown tools, 'Wayfinders' and 'Room Tools,' to address challenges in navigating Facebook's offices and managing meeting spaces. It highlights the dynamic updating of digital maps, real-time conference room availability displays, on-the-fly booking capabilities, and issue reporting. Furthermore, it describes the integration of motion sensors and calendar data to optimize conference room utilization by identifying and reclaiming unused booked rooms, leading to significant reclaimed meeting room inventory and data-driven insights into work habits and space utilization.

6/25/2015

This post details the rapid development of the 'add a link' feature for iPhone, highlighting how an initial user frustration was addressed through a hackathon. It showcases the process of moving from a functional prototype to a product, involving cross-functional collaboration, user research, and data-driven experimentation. The post emphasizes the iterative nature of development, including real-time parameter adjustments and bakeoffs, and the importance of understanding user behavior through data analysis post-pilot.

6/19/2015

This post introduces Folly Futures, an implementation of futures for C++11 used at Facebook. It explains the benefits of asynchronous programming over synchronous programming for improving efficiency and reducing resource waste. The post details how futures simplify complex asynchronous operations, enabling sequential and parallel composition of computations, and providing cleaner error handling compared to traditional callback-based approaches. It also touches upon execution context control and the use of Promises for creating futures, highlighting how these advancements contribute to more robust and readable code for internal services.

4/10/2015

This post introduces CT-Scan, a performance monitoring and prediction platform developed to address the challenge of regressions in complex mobile apps. It details the system's design principles focused on pattern detection and diagnostics across development, staging, and production lifecycles. The post highlights the system's ability to run automated experiments, gather detailed diagnostic data, and sample real-world performance counters, all while aiming to maintain or improve development speed and minimize performance regressions in speed, data usage, battery consumption, and memory footprint. It also touches upon the scalability and extensibility of the system, including the development of a device lab.

12/30/2014

This post details the 2014 hackathon initiatives, showcasing how these events foster innovation across product, infrastructure, data, and culture. It highlights specific projects like Autoscale for energy efficiency, experimental drone hacks, the @Scale app, and osquery for infrastructure insight, demonstrating the practical and experimental outcomes of hackathons in driving new ideas and community building.

10/2/2014

This post highlights Meta's focus on mobile development challenges and solutions presented at @Scale 2014. It details advancements in iOS architecture (custom persistence layer, Components view architecture), cross-platform C++ development (Dropbox's use of C++, Djinni tool), Android development (Buck, Exopackage for faster builds), and scaling apps for emerging markets. It underscores the company's commitment to rapid iteration, efficient tooling, and addressing the unique demands of the mobile ecosystem.

10/1/2014

This post introduces Meta's refined approach to configuration management using Chef, emphasizing a data-driven API for managing server configurations. It details a new model for writing community-style cookbooks that are more flexible and maintainable by treating configuration files as data structures. The post also highlights the open-sourcing of specific cookbooks, documentation, and tools like Grocery Delivery and a new testing system, aiming to foster community contribution and adoption of their practices.

9/20/2012

This post details the adaptation of Facebook's established web release process to mobile applications. It introduces a date-driven release model for mobile apps (Android, iOS, Messenger, Camera) to ensure predictable, regular updates (4-8 week cycles). The post highlights the unique challenges of mobile releases, emphasizing a higher quality bar, the need for targeted fixes, and the importance of code reuse across platforms and standalone apps. It marks a significant step in formalizing and scaling the mobile development and deployment pipeline.

8/8/2012

This post details Facebook's approach to product testing and launching, emphasizing continuous iteration and data-driven decision-making. It introduces the concept of 'shipping early, shipping often' and highlights the use of a sophisticated tool called 'Gatekeeper' for managing A/B tests. The post provides an example of a failed feature launch (chat bar) and the subsequent rapid iteration based on measured impact, demonstrating the importance of empirical data over internal debate. It also showcases a successful small tweak that led to a significant increase in friend requests by optimizing the 'find friends' interface.

8/3/2012

This post details the decision to double the release frequency of facebook.com to twice daily, driven by the growth of the engineering team to 1000 developers and a 6x increase in code production per week. This change aims to maintain the rapid and efficient release process established when the team was smaller, ensuring engineers globally have equal opportunities to ship code quickly.

5/15/2012

This post highlights the importance of a robust developer toolchain and build system (fbconfig) in enabling engineers to move fast and focus on shipping new products. It emphasizes the impact of improving these internal tools on overall engineering productivity and the ability to iterate quickly on core product features like network performance and load balancing. The post also touches upon the lightweight development process and the culture of shared ownership that encourages engineers to improve any part of the codebase, including the toolchain.

4/5/2012

This post details the critical role of release engineering in enabling Facebook's rapid development and deployment culture. It highlights the establishment of daily pushes, the development of tools and processes to manage thousands of changes weekly across hundreds of engineers, and the introduction of a 'push karma' system to balance speed with risk. The post emphasizes the importance of engineers taking end-to-end responsibility for their code and the continuous effort to 'move faster' while maintaining site reliability and user experience.

3/7/2012

This post highlights Meta's culture of encouraging risk-taking and rapid iteration in feature development, exemplified by the mantra 'Move Fast and Be Bold.' It details how research ideas can be quickly prototyped and deployed to users, with a focus on failing early and iterating rapidly. The post also emphasizes the importance of experimentation and measurement through systems like Gatekeeper and the use of Hadoop for data analysis, reinforcing the scientific approach to product development.

9/30/2011

This post details the evolution of Facebook's browser testing frameworks, starting with the adoption of Watir in early 2009 and its subsequent integration into critical workflows like 'push blocking'. It highlights the challenges faced with Watir, such as its JavaScript dependency and lack of integration with the dominant PHP unit test system. The post then describes the transition to WebDriver, emphasizing its language-agnostic nature and the successful creation of a PHP WebDriver client. This shift aimed to improve test reliability, speed up feedback loops, and better leverage existing engineering investments, reflecting Meta's culture of embracing better alternatives for product quality.

9/27/2011

This post introduces a 'Software Design Glossary' that defines key concepts like Element, Coupling, Cohesion, Succession, Socialization, Latency, Throughput, and Variance. It provides engineers with a shared vocabulary to discuss and implement robust software design, emphasizing the importance of managing dependencies (coupling), modularity (cohesion), safe change implementation (succession), community absorption of change (socialization), and balancing speed with predictability (latency, throughput, variance). This contributes to the overall goal of empowering engineers with the tools and understanding for rapid and high-quality feature development.

2/15/2011

This post highlights a successful hackathon project that led to a complete redesign and rebuild of the internal Pixelcloud tool. The redesign focused on improving the user interface, enhancing search capabilities, and optimizing backend storage and database queries, resulting in a significant increase in image uploads and comments. It demonstrates how hackathons foster rapid innovation and iterative improvement of internal tools.

1/26/2011

This post details the creation and release of JSGameBench, a new benchmark suite developed during a hackathon to measure and improve HTML5 game performance, specifically focusing on sprite rendering. It highlights the potential of HTML5 for web and mobile gaming, analyzes current browser performance limitations, and outlines future development directions for HTML5 game engines and browser technologies.

2/11/2010

Introduced an internal dashboards framework to address the challenge of numerous custom internal tool requests. The framework allows non-engineers to build dynamic custom pages in the browser without coding and enables engineers to break down complex tools into reusable, configurable widgets. Defined the concept of a widget as a child object of a main widget class with well-defined parameters and HTML rendering based on configuration. Described the architecture at a high level with a code example for a widget and highlighted the ability for viewer-configurable dashboards.

7/30/2009

This post contributes by detailing research into how popularity spreads on Facebook, specifically analyzing the network effects and social dynamics behind the growth of Facebook Pages. It highlights how the News Feed facilitates the spread of information and how interconnected communities of users drive adoption, challenging the idea of individual influencers being the primary drivers of popularity. This research provides insights into user engagement and feature adoption patterns.

3/23/2009

This post describes Facebook's unique development and deployment environment where engineers are deeply involved in the release process, enabling changes to go from code submission to production in under 60 minutes. It emphasizes the trade-off of engineer time and attention for this rapid iteration capability.