Strobelight: A profiling service built on open source technology

Introduced Strobelight, Meta's profiling orchestrator, which combines multiple profilers (many open source, often using eBPF) to collect detailed performance metrics (CPU usage, memory allocations, etc.) from all production hosts. Strobelight enables engineers to identify performance bottlenecks, optimize code, and improve utilization, leading to significant efficiency wins (e.g., 15,000 servers' worth of annual capacity savings). It supports 42 different profilers, including memory, function call count, event-based, AI/GPU, off-CPU time, and request latency profilers. Strobelight allows for on-demand profiling via CLI/UI and continuous/triggered profiling via configuration files. It includes safeguards against performance degradation and profiler conflicts. A core principle is providing automatic, regularly-collected profiling data for all services, with dynamic sampling and weighted samples for aggregation. The LBR profiler feeds into FDO pipelines for binary optimization, and the event profiler collects stack traces for monitoring and testing. Stack Schemas enhance data with tags for better analysis.