Autonomous testing of services at scale

This post details the design and implementation of an autonomous testing extension built on top of Meta's existing integration testing infrastructure for backend services. Key contributions include: 1. Reusing production infrastructure (containerization, routing) to build ephemeral, isolated test environments. 2. Implementing a nuanced isolation strategy that allows known read-only traffic, service owner-defined allowlists, rerouting to universal mocks, and blocking all other network requests. 3. Utilizing application-level (API-based) and network-level (IP:port-based) isolation, with stack trace inspection for code-level decisions. 4. Employing LD_PRELOAD over BPF for the isolation layer implementation. 5. Demonstrating how this system enables autonomous testing for one-third of Meta's services.
More from the Site Reliability and Outage Management trail
- Lights Out, Systems On: Validating Instant Power Loss Readiness
- DrP: Meta’s Root Cause Analysis Platform at Scale
- Zoomer: Powering AI Performance at Meta’s Scale Through Intelligent Debugging and Optimization
- Diff Risk Score: AI-driven risk-aware software development
- How Meta keeps its AI hardware reliable