TechPrufer
Get started
Meta
Meta logoMeta · Feature trail

Agentic Development and Just-in-Time Testing

5 posts · Jun 2017 – Sep 2026

What this trail covers

Meta is exploring agentic development and introducing Just-in-Time Tests (JiTTests) as a novel approach to software testing. JiTTests leverage LLMs to automatically generate tests on-the-fly for specific code changes, aiming to catch bugs just before they enter production. This approach contrasts with traditional testing methods that rely on manually authored, maintained, and executed test suites. The JiTTest process involves inferring code change intent, creating mutants (code with deliberate f. This post details the Automated Compliance Hardening (ACH) tool, which uses mutation-guided, LLM-based test generation. ACH hardens platforms against regressions by generating undetected faults (mutants) in source code specific to an area of concern and using those same mutants to generate tests. ACH automatically generates unit tests that target a particular kind of fault, described in plain text, and provides verifiable assurances that its tests catch the described faults. This approach is a departure from traditional methods that focus on increasing code coverage, instead targeting specific faults. ACH has been applied to platforms like Facebook Feed, Instagram, Messenger, and WhatsApp.

All Meta trails
  1. An Organizational Second Brain: Building an AI That Learns From ExpertsThis post details the architecture and implementation of an AI agent designed to act as an organizational "second brain" for specialist knowledge. Key contributions include: a four-layer architecture (knowledge, reasoning, evaluation, improvement loop), a knowledge system structured into "position files," "taxonomy and vocabulary files," "routing indexes," and "gateway files" with explicit dependencies, a split between high-density wiki knowledge and sparse RAG retrieval, and "composable recipes" for imperative reasoning that separate knowledge from procedure. It also introduces "checkpoints" and "escalations" to keep human experts in control.
  2. Capacity Efficiency at Meta: How Unified AI Agents Optimize Performance at HyperscaleThis post details the development of a unified AI agent platform for capacity efficiency at Meta. It describes how standardized tool interfaces and encoded domain expertise (skills) are used to automate both finding (offense) and fixing (defense) performance issues across hyperscale infrastructure. Key contributions include the AI Regression Solver, which automatically generates pull requests to fix regressions, and an AI-assisted opportunity resolution system for proactive optimizations. The platform leverages shared tools and data sources, enabling compounding returns and expanding to new applications like conversational assistants and capacity planning agents.
  3. How Meta Used AI to Map Tribal Knowledge in Large-Scale Data PipelinesDeveloped a pre-compute engine composed of 50+ specialized AI agents to systematically read codebase files and generate 59 concise context files encoding tribal knowledge. This system provides structured navigation guides for 100% of code modules, documents 50+ non-obvious patterns, and reduces AI agent tool calls by 40% per task. Implemented a self-refreshing mechanism for context files and generated cross-repo dependency indexes and data flow maps.
  4. Ranking Engineer Agent (REA): The Autonomous AI Agent Accelerating Meta’s Ads Ranking InnovationIntroduced the Ranking Engineer Agent (REA), an autonomous AI agent designed to drive the end-to-end ML lifecycle for ads ranking models. REA features a hibernate-and-wake mechanism for long-horizon autonomy, a dual-source hypothesis engine combining historical insights and an ML research agent, and a three-phase planning framework (Validation, Combination, Exploitation) for resilient execution within compute budgets. It achieved 2x model accuracy and 5x engineering productivity gains in its initial rollout.
  5. Patch Me If You Can: AI Codemods for Secure-by-Default Android AppsThis post describes the development of a system that uses generative AI to automate the migration of existing Android code to secure-by-default frameworks. The system can propose, validate, and submit security patches at scale, reducing friction for engineers.
  7. The Death of Traditional Testing: Agentic Development Broke a 50-Year-Old Field, JiTTesting Can Revive ItThis post introduces Just-in-Time Tests (JiTTests) as a novel approach to software testing, leveraging LLMs to automatically generate tests on-the-fly for specific code changes. The process involves inferring code change intent, creating mutants (code with deliberate faults), and generating/running tests to catch those faults. Ensembles of rule-based and LLM-based assessors focus the signal on true positive failures, providing engineers with clear, relevant reports about unexpected behavior changes. This eliminates ongoing maintenance costs, reduces false positives, and allows testing to keep pace with agentic coding.
  8. LLMs Are the Key to Mutation Testing and Better ComplianceThis post details Meta's Automated Compliance Hardening (ACH) tool, which uses LLMs to address the challenges of scaling mutation testing. ACH generates realistic, problem-specific mutants and corresponding tests, overcoming issues of scalability, unrealistic mutants, equivalent mutants, computational cost, and overstretching testing efforts. It leverages LLMs for mutant generation, equivalence detection, and test creation, with a trial demonstrating high acceptance rates for generated tests and their utility in catching previously undetected faults.
  9. Diff Risk Score: AI-driven risk-aware software developmentIntroduces Diff Risk Score (DRS), an AI-powered system using a fine-tuned Llama LLM to predict production incident risk from code changes. DRS evaluates code and metadata to provide a risk score and highlight risky snippets, enabling features like code unfreezes during sensitive periods. Future work includes predicting configuration change risk, automating risk mitigation with AI agents, and improving explainability through natural language outputs.
  10. Introducing AutoPatchBench: A Benchmark for AI-Powered Security FixesIntroduced AutoPatchBench, a benchmark for evaluating AI program repair systems specifically for fuzzing-found vulnerabilities. This benchmark includes 136 C/C++ vulnerabilities with verified fixes from the ARVO dataset, curated for reproducibility and with automated verification mechanisms including fuzzing and white-box differential testing.
  11. Revolutionizing software testing: Introducing LLM-powered bug catchersIntroduced the Automated Compliance Hardening (ACH) tool, a system for mutation-guided, LLM-based test generation. ACH generates undetected faults (mutants) in source code and uses them to generate tests that are guaranteed to catch bugs. The system takes plain text descriptions of faults and automatically generates relevant mutants and tests. This approach is a departure from traditional test generation that focuses on code coverage, instead targeting specific faults. ACH has been applied to various Meta platforms including Facebook Feed, Instagram, Messenger, and WhatsApp.
  12. Inside Bento: Jupyter Notebooks at MetaThis post introduces Bento, Meta's internal distribution of Jupyter Notebooks. It highlights the platform's ability to mix code, text, and multimedia for various use cases, including prototyping and ML workflows. Key features discussed are scheduled notebooks, sharing capabilities, and serverless notebook execution leveraging WebAssembly in the browser.
  13. How do you test your tests?Introduced the Probabilistic Flakiness Score (PFS) to quantify test flakiness. Developed a statistical model using Bayesian inference to estimate PFS based on observed test results. Incorporated an empirical observation of developer asymmetry in treating passing vs. failing tests into the PFS calculation.
  14. Getafix: How Facebook tools learn to fix bugs automaticallyGetafix introduces a novel approach to automated bug fixing by learning from past code changes. It utilizes hierarchical clustering and anti-unification to mine abstract fix patterns, considering both code edits and their surrounding context. This allows for more general and context-dependent fixes than simpler auto-fix tools. The system employs a tree differencer to identify concrete edits and a greedy clustering algorithm enhanced with contextual information to discover reusable patterns. Getafix then uses these patterns to generate and rank potential patches for new bugs.
  15. Training AI agents to solve unfamiliar tasksIntroduced composable planning for AI agents, enabling them to solve unfamiliar tasks by learning a model of their environment based on attributes and performing a range of basic tasks. This allows agents to plan and execute longer, more complex tasks by combining simpler ones.
  16. Finding and fixing software bugs automatically with SapFix and SapienzThis post introduces SapFix, an AI hybrid tool that automatically generates and proposes fixes for software bugs. It leverages strategies like code reversion, templated fixes, and mutation-based AST modifications. SapFix autonomously validates generated fixes by checking for compilation errors, persistent crashes, and new regressions using existing tests and Sapienz-generated tests. Approved fixes are then presented to human reviewers for deployment. This represents a significant advancement in automating the debugging process at scale.
  17. Talk the Walk: Teaching AI systems to navigate New York through languageIntroduced the 'Talk the Walk' research task for embodied AI, training agents to navigate real NYC neighborhoods through communication. Developed the MASC (Masked Attention for Spatial Convolution) attention mechanism to improve the guide bot's map focus. Explored emergent communication protocols (continuous vectors, synthetic language) and compared AI performance against human players, demonstrating the challenges of natural language grounding in complex visual environments.
  18. Embodied Question Answering: A goal-driven approach to autonomous agentsIntroduced Embodied Question Answering (EmbodiedQA) as a new AI task and dataset. Developed House3D, a collection of 45,000 simulated indoor environments for training and testing autonomous agents. Designed AI agents capable of perception, navigation (planner-controller module), and communication to answer questions in natural language within these environments. Explored learning paradigms including active perception, commonsense reasoning, language grounding, and credit assignment using reinforcement learning, imitation learning, and reward shaping.
  19. Sapienz: Intelligent automated software testing at scaleSapienz is an intelligent automated software testing tool that leverages search-based software testing to design and execute test cases through the UI. It builds a model of the system under test by sampling the space of possible tests, combining individual events with motif events. Sapienz has been deployed at scale for Facebook's Android app, automating tens of thousands of test cases daily, resulting in faster bug fixes and a low false-positive rate. It integrates with continuous integration systems and reports actionable issues.
  20. Deal or no deal? Training AI bots to negotiateIntroduced dialog agents capable of negotiation, trained on a multi-issue bargaining task. Developed 'dialog rollouts' for simulating future conversations to select optimal utterances. Agents learned to negotiate harder, employ tactics like feigning interest, and generate novel sentences. Trained via imitation learning on human data and fine-tuned with reinforcement learning, achieving human-level negotiation performance.