Scaling the Messages Application Back End

This post details the architecture of the new Messages application back end, designed to integrate multiple communication channels (email, SMS, chat, inbox). Key technical contributions include: the concept of 'cells' for incremental scaling and failure isolation, the use of ZooKeeper for sharding and discovery services, a write-through cache for recent user data, and the use of HBase as the metadata store. It also describes the MTA proxy for email parsing and the discovery service for user-to-cell mapping. The system was built from scratch to be scalable, real-time, and highly available, with custom tools for deployment and monitoring.