Keeping the Site Reliable While Moving Fast

This post details the initial efforts in measuring and monitoring user-visible errors across server and client sides. It describes logging errors with Facebook Scribe (1% sampling rate), including details like browser type, user country, and stack traces. The massive data (600GB/day) is processed using Apache Hive to generate trends and compute reliability metrics such as error-free page requests and successful service requests. Specific examples of uncovered issues and their fixes are provided, including client-side JavaScript event alignment, Chat connection and message sending problems, database access errors due to lack of local replicas, and back-end service robustness improvements through throttling and rate limiting. The post also outlines the development of a real-time monitoring system with over 1500 trend lines and the use of machine learning for anomaly detection and alarm generation.