Building Realtime Insights

This post details the engineering of a real-time insights system for Facebook Insights for Websites. It describes the challenges of processing 20 billion events per day with sub-30-second latency, including the failure of MySQL counters and in-memory counters due to write rate and reliability issues, respectively. The chosen architecture utilizes HBase for its high write rate and reliability, scribe for log aggregation built on HDFS, and a client for batching and writing data streams from scribe to HBase. Key architectural considerations include table schema, key composition, batching nuances, and sharding.