Keeping the Site Reliable While Moving Fast

What this post added

This post details the evolution of Facebook's site reliability efforts, focusing on the development of a comprehensive error logging and monitoring system. It describes the implementation of server-side and client-side error logging using Scribe, the processing of massive error data with Apache Hive for trend analysis, and the calculation of reliability metrics. The post highlights the identification and fixing of various user-visible errors, including issues in News Feed comments, Chat presence servers, database access, and back-end services. It also outlines the creation of a real-time monitoring system with over 1500 trend lines and the use of machine learning for anomaly detection and alarm generation, which has resulted in a 5x reduction in user-visible errors.