TechPrufer
Get started
Meta
Meta logoMeta · Feature trail

Facebook Chat

28 posts · May 2008 – Mar 2024

What this trail covers

Meta's real-time communication capabilities have evolved significantly, starting with the foundational challenges of providing real-time messaging and presence notifications with Facebook Chat. This involved overcoming issues with handling large-scale presence updates, ensuring timely message delivery through techniques like XHR long polling, and building fault-tolerant distributed systems. The 'mercury project' introduced modular design, test-oriented development, and functional programming to the backend. The Messages application back end was built from scratch to integrate email, SMS, Facebook Chat, and the existing Facebook Inbox, requiring a new infrastructure stack. This stack features application servers with an API entry point, distributed logic managed by cells and ZooKeeper for sharding and failover, a write-through cache, and a data access layer with a time-sequenced log and snapshots backed by HBase. Other services include an MTA proxy for email parsing, a discovery service for user-to-cell mapping, and a user index service for social information. Tools for daily deployments, monitoring, and performance analysis were also developed.

All Meta trails
  1. Making messaging interoperability with third parties safe for users in EuropeThis post details the technical implementation of messaging interoperability for WhatsApp and Messenger to comply with the EU's Digital Markets Act (DMA). It describes the use of the Signal Protocol for end-to-end encryption, the construction of message protobuf structures encrypted and packaged into XML stanzas, and the client/server architecture for interoperation. Key technical aspects include the use of XMPP for client-to-server communication, HTTP for interfacing with third-party servers, OpenID and JWT for user identifier authentication, and the Noise Protocol Framework for client-to-server encryption. It also discusses the trade-offs and challenges of not having direct control over both sending and receiving clients, and explores an alternative proxy-based approach.
  2. Threads: The inside story of Meta’s newest social appThis post details the rapid development and launch of Meta's newest social app, Threads. It highlights the startup-like environment, the use of existing technologies like Python (Django) and Instagram's infrastructure, and the ambitious goal of creating a decentralized and interoperable social platform. The post emphasizes the speed of development, the successful scaling to over 100 million users, and the future vision of integrating with protocols like ActivityPub to achieve email-like interoperability.
  3. Launching Instagram Messaging on desktopThis post details the engineering effort to launch Instagram Messaging on desktop web. Key technical contributions include adapting the existing mobile web messaging architecture (built with React hooks, RESTful endpoints, MQTT, and ServiceWorkers) for a desktop environment. This involved improving UX with desktop-centric actions like drag-and-drop, adding features like presence and voice messages, supporting business accounts, and enhancing caching mechanisms for a more up-to-date experience. The project leveraged code reusability from the mobile web implementation and involved significant A/B testing to validate its impact on user engagement.
  4. How WhatsApp enables multi-device capabilityThis post details the technical architecture and security considerations for WhatsApp's new multi-device capability. It explains how WhatsApp moved from a smartphone-centric architecture to a standalone multi-device experience, addressing challenges in maintaining end-to-end encryption, synchronizing data across devices, and adapting voice/video protocols. Key technical contributions include the client-fanout approach for message delivery, new technologies for device identity management and verification, and methods for synchronizing message history and application states while preserving privacy.
  5. A smaller, faster video calling library for our appsIntroduced 'rsys', a new, smaller, and faster cross-platform video calling library for Meta's apps. Rewritten from scratch using the latest WebRTC, rsys is approximately 20% smaller than the previous library, supports multiple platforms (Android, iOS, MacOS, Windows, Linux), and features a modular architecture with pluggable components and a Flux-based SDK for extensibility. Key technical improvements include a unified state machine architecture for signaling and media stacks, abstraction of OS-dependent functionality, and optimization of binary size through selective compilation and use of optimally sized libraries.
  7. Co-experiences: Hanging out in the digital living roomThis post details the development of 'co-experiences,' specifically focusing on the 'Watch Together' feature. It introduces a new state synchronization platform built on top of existing multiway infrastructure. This platform uses Thrift structures for input and output, and its three main parts are subscription, input, and resolution. The platform centralizes resolution logic on the server to save client processing power and maintain consistency, supporting web, Android, and iOS across Messenger and Instagram. It also powers other features like screen-sharing moderation and Rooms to Live.
  8. Project LightSpeed: Rewriting the Messenger codebase for a faster, smaller, and simpler messaging appThis post details Project LightSpeed, a massive undertaking to rewrite the Messenger iOS codebase from scratch. It highlights the achievement of making the app twice as fast and one-fourth the size by reducing core code by 84%. Key technical contributions include leveraging native OS features for rendering and threading, using dynamic UI templates powered by SQLite for code reuse, and implementing a server broker for efficient communication. This represents a significant architectural shift to improve performance, reduce complexity, and ensure long-term scalability for Messenger.
  9. Facebook releases Safety Check crisis response tool for WorkplaceThis post details the adaptation of Facebook's Safety Check feature for the Workplace platform. Key technical contributions include migrating storage to TAO for enterprise scale, developing a custom access management module for enhanced privacy, implementing a location parser to standardize company-specific location data, and designing a multi-channel notification system with per-person job handling and retry logic for reliability. The 'iterate' phase utilizes graph storage for real-time status updates and a queuing system to manage response volume, leveraging GraphQL subscriptions for real-time updates.
  10. Migrating Messenger storage to optimize performanceThis post details the significant overhaul and modernization of Messenger's storage service to improve performance, efficiency, reliability, and ease of upgrades. Key contributions include the migration from HBase to MyRocks (integrating RocksDB as a MySQL storage engine) and from spinning disks to flash storage on Lightning Server SKUs. It also elaborates on the complex, two-phase migration process (normal and buffered flows) designed to handle over a billion users without disruption, including data schema redesign, invariant enforcement, and extensive data validation. The post highlights the resulting 90% reduction in storage consumption, 50x lower read latency, and enhanced disaster recovery automation.
  11. Building Mobile-First Infrastructure for MessengerIntroduced a new mobile-first sync protocol for Messenger, moving from a pull-based model to a push-based snapshot + delta model using MQTT. Replaced JSON with Thrift for message and delta updates, reducing payload size by ~50%. Developed a new server-side service called Iris, which acts as a totally ordered queue of messaging updates, enabling independent synchronization rates for the Messenger app and traditional storage tiers. Implemented a tiered storage model with Iris's memory, backing store, and traditional disk storage.
  12. Building a faster MessengerThis post details the technical efforts to significantly improve the startup time and overall performance of the Messenger app on both iOS and Android. On iOS, a custom wall-clock tracing utility (FBTracer) was developed to identify and eliminate non-essential startup processes. On Android, bytecode manipulation using ASM was employed to optimize the dependency injection framework, achieving a 30% speed gain. Further performance enhancements included view controller reuse and backgrounding sound playback on iOS, and optimizing the contact schema and queries on Android. Automated diff testing for startup timing was implemented to catch regressions.
  13. Under the hood: Facebook Messenger for FirefoxThis post details the integration of Facebook Messenger into Firefox 17, leveraging HTML5 WebSockets and Shared Workers. It highlights the use of WebSockets for efficient, persistent client-server connections and the MQTT protocol for bandwidth optimization. Shared Workers are used to synchronize state across various Facebook elements (notifications, chat tabs, etc.) without redundant data fetching. The post also discusses the challenge of reusing existing JavaScript libraries within a DOM-less Shared Worker context and the solution of adapting these libraries to remove DOM dependencies.
  14. Under the Hood: Integrating standalone mobile app features into Facebook for iOSThis post details the technical and organizational evolution of integrating standalone mobile app features (specifically Camera and Messenger) into the core Facebook for iOS application. It highlights the shift to a unified product team model responsible for cross-platform experiences, the use of native code and timed release cycles for faster development and deployment, and the strategic decision-making process for feature integration based on user experience and added value to the core app. It also mentions the technical implementation of integrating Messenger's chat access (left swipe) and Camera's multi-photo selection into the core iOS app.
  15. Under the hood: Improving messagesThe 'mercury project' significantly improved the reliability and development process for Facebook messages and chat. Key contributions include: adoption of a CommonJS-like module system to replace a fragile JavaScript dependency management system, leading to better code organization and reduced regressions; emphasis on test-oriented development with over 85% test coverage and a culture of accepting patches only with accompanying tests; and a shift from Object Oriented Programming (OOP) patterns to functional programming, modeling domain objects as structured data rather than class prototypes. This functional approach enables fast, incremental loading and updates, making the system more real-time and easier to cache.
  16. Under the Hood: Building Facebook Messenger for WindowsThis post details the development of Facebook Messenger for Windows, a desktop application for real-time chat and notifications. Key technical contributions include: using HTML and JavaScript for the client-side UI and logic, hosted within a web browser control, allowing for server-side updates; employing MQTT for reliable, low-overhead, persistent message delivery; and utilizing a meta-installer and Google's Omaha for automated client updates. The design prioritized a native look and feel while maintaining consistency with the Facebook web UI.
  17. Interning at Facebook: Who Goes Where When (and Why It Matters)This post details how check-in data, a real-time user interaction feature, was leveraged to develop algorithms for predicting business operating hours. It also describes the implementation of crowd-sourcing tools and validation processes to gather and refine this information, ultimately improving the user experience by making it easier to find open businesses.
  18. Building Facebook MessengerThis post details the technical challenges and solutions involved in building Facebook Messenger as a standalone application. Key contributions include the adoption of MQTT for persistent, low-latency mobile connections to reduce message delivery time from seconds to hundreds of milliseconds, and the integration of different communication channels (chat, text, email) to create a seamless user experience. It also highlights the challenges of adapting to the existing Facebook Messages infrastructure and making product decisions around communication channel integration.
  19. Building Video CallingThis post details the initial development of Facebook's video calling feature, focusing on overcoming technical challenges related to installation flows across different browsers and operating systems, protecting user identity through secure API integration with Skype, and managing complex client-side state in a plugin environment. It highlights the need to support a wide range of user environments and browser behaviors, including handling asynchronous interactions and surviving page reloads after plugin installation. The author also mentions the use of HTML5 features like sessionStorage, which were omitted for broader compatibility.
  20. Building the Send ButtonIntroduced the Send button, a new social plugin that allows users to send links to a subset of their friends, complementing the Like button. Implemented a dialog (flyout) that communicates with the button via the Facebook Connect JavaScript SDK. Developed a custom typeahead for recipient suggestions, including friends, groups, and external email addresses, by extending core Facebook tokenizer and typeahead modules. Leveraged the Messages system for sending to various recipient types. Enabled combining Like and Send buttons in a single XFBML tag.
  21. Inside Facebook Messages’ Application ServerThis post details the internal architecture of the Facebook Messages application server, a dedicated system built to manage the infrastructure for Facebook Messages. It describes the business logic for creating, reading, deleting, and updating messages, including handling email, SMS, and chat integrations. Key technical aspects include the use of HBase for metadata storage with a denormalized schema for message threads, caching of user metadata using LRU caches to reduce HBase queries, and synchronization mechanisms to handle concurrent updates. It also covers the storage schema, metadata entities and indexes, action logs for state management and schema evolution, and real-time search indexing using Apache Lucene. The post highlights the use of dark launches for testing and validation.
  22. Scaling the Messages Application Back EndThis post details the architecture of the new Messages application back end, designed to integrate multiple communication channels (email, SMS, chat, inbox). Key technical contributions include: the concept of 'cells' for incremental scaling and failure isolation, the use of ZooKeeper for sharding and discovery services, a write-through cache for recent user data, and the use of HBase as the metadata store. It also describes the MTA proxy for email parsing and the discovery service for user-to-cell mapping. The system was built from scratch to be scalable, real-time, and highly available, with custom tools for deployment and monitoring.
  23. Live Commenting: Behind the ScenesThis post details the engineering challenges and solutions for implementing 'live commenting' on Facebook. It describes the shift from a polling-based approach to a push-based system to achieve real-time communication. Key technical contributions include the design of a system to track viewer-to-content associations at scale (16 million new associations per second) and the adoption of a "write locally, read globally" storage strategy to manage high write loads and reduce cross-country bandwidth usage.
  24. The Underlying Technology of MessagesThis post details the technology behind a new version of Facebook Messages that unifies chat, SMS, and email into a real-time conversation. The infrastructure handles over 350 million users and 15 billion messages per month. The team evaluated MySQL, Cassandra, and HBase for storage, ultimately choosing HBase due to its scalability, performance, and simpler consistency model for handling both volatile and long-tail data. They also developed a new application server from scratch to interface with various services like Haystack for attachments and ZooKeeper for user discovery, ensuring a robust and performant real-time messaging system.
  25. Chat reaches 1 billion messages sent per dayThis post details reaching 1 billion messages per day for Facebook Chat. It describes specific engineering improvements made to handle this scale, including increasing load balancer capacity, investigating backend/frontend robustness against failures, and optimizing channel servers. A key technical detail is the reduction of memory load by explicitly discarding unused HTTP request headers (cookies and other headers) after they are no longer needed for response construction, which frees up memory to serve more users per machine.
  26. Chat Stability and ScalabilityThis post details the engineering challenges and solutions for scaling Facebook Chat to millions of users. It discusses the use of Erlang for its concurrency and distribution features, optimizations for memory footprint (e.g., using character arrays instead of linked lists, forced garbage collection before sleeping), and debugging C++ chatloggers that suffered from heap fragmentation due to excessive allocations from lexical_cast. It also covers the presence servers' bandwidth challenges, leading to zlib compression, and operational issues with load balancers causing connection resets and outages. The post highlights the addition of more load balancers as a solution and the implementation of monitoring tools like Scribe and fb303 for system health.
  27. Thrift: (slightly more than) one year laterThis post provides an update on the Thrift framework, highlighting its significant evolution in its first year as an open-source project. It details the addition of new language support (including Erlang, which was previously adopted for Facebook Chat), protocol enhancements, performance improvements, and the impact of external contributions. The post also discusses ongoing development at Facebook (asynchronous C++ server) and within the community (Ruby mapping improvements, C mapping). The acceptance into the Apache Incubator is presented as a major milestone for Thrift's future growth and integration. This post demonstrates Thrift's increasing maturity and its role as a foundational technology for inter-service communication at Facebook, directly impacting the scalability and development of systems like Facebook Chat.
  28. Facebook ChatThis post details the engineering challenges and solutions for building Facebook Chat, focusing on real-time presence notification and messaging. It highlights the inefficiency of naive presence updates and introduces XHR long polling (a variation of Comet/BOSH) for real-time messaging. The post also discusses distribution, isolation, and failover strategies, including the use of C++ for logging and Erlang for an epoll-driven web server to handle in-memory conversations and long-polled requests. Thrift is presented as the glue for cross-language communication between PHP, JavaScript, Erlang, and C++.