Internal Load Balancing Service

6 posts · Sep 2017 – Oct 2024

What this trail covers

Dropbox's internal load balancing service, Robinhood, has evolved to address uneven load distribution and improve reliability. The latest iteration leverages PID controllers for more effective load management, leading to significant hardware cost savings and better handling of AI workloads. The system architecture includes a load balancing service, proxy, and routing database, with detailed strategies for handling LBS restarts, cold start nodes, and missing load reports. It supports load balancing based on CPU utilization and/or in-flight requests. Cross-data center routing is handled via locality configuration.

All Dropbox trails

Oct 2024What’s new with Robinhood, our in-house load balancing serviceThis post details the latest iteration of Dropbox's internal load balancing service, Robinhood. Key technical contributions include the integration of PID controllers for more effective load management, leading to improved reliability and significant hardware cost savings. The post also elaborates on the architecture, including the load balancing service (LBS), proxy, and routing database (ZooKeeper/etcd-based). It details strategies for handling LBS restarts, cold start nodes, and missing load reports, and supports load balancing based on CPU utilization and/or in-flight requests. Cross-data center routing considerations and locality configuration are also discussed.
Mar 2021Atlas: Our journey from a Python monolith to a managed platformThis post details the journey from a monolithic Python server (Metaserver) to a managed platform called Atlas. Atlas aims to provide the benefits of a Service Oriented Architecture (SOA) with minimized operational costs for self-contained functionalities. It abstracts away infrastructure management for developers, allowing them to focus on writing endpoint logic. The migration involved standardizing on gRPC and Envoy, and introducing autoscaling and canary analysis. The post contrasts the developer experience before and after Atlas, highlighting improvements in code structure, independent and consistent pushes, and reduced infrastructure debt. Atlas coexists with larger, independently operated services.
Jul 2020How we migrated Dropbox from Nginx to EnvoyThis post details the migration of Dropbox's traffic infrastructure from Nginx to Envoy. It highlights the limitations of the legacy Nginx setup, including its static configuration, reliance on Lua for dynamic logic, and operational maintenance costs. The post compares Nginx and Envoy across performance, observability, and configuration dimensions, concluding that Envoy offers better performance, especially with stats collection, and a more modern, flexible configuration model. The migration to Envoy enabled significant server cost savings and improved handling of gRPC and other modern protocols.
Sep 2019Enhancing Bandaid load balancing at Dropbox by leveraging real-time backend server load informationThis post details the enhancement of Dropbox's Bandaid layer-7 load balancing service by incorporating real-time backend server load information. The approach involves piggybacking server utilization (active requests / max concurrent requests) in HTTP response headers (X-Bandaid-Utilization). This allows Bandaid instances to make more informed load balancing decisions using a 'random N choices' method, selecting servers based on their computed scores derived from this utilization metric. The design addresses challenges of distributed load balancing in cloud environments by leveraging readily available application-layer metrics.
Mar 2018Meet Bandaid, the Dropbox service proxyThis post introduces Bandaid, a service proxy written in Go, designed to accelerate the adoption of Service Oriented Architecture at Dropbox. Bandaid functions as a reverse proxy with advanced features including various load balancing methods (round-robin, least N random choices, absolute least connection, pinning peer), SSL termination, HTTP2 support, metro rerouting, request/response buffering, logical isolation of endpoints, dynamic reconfiguration, service discovery, per-route stats, gRPC proxying, health checking, and weighted traffic management for canary testing. Key internal design aspects include a request handler that routes requests to queues, which are then processed by a fixed-size worker pool. The queueing mechanism uses LIFO processing to manage overloaded backends and prevent processing of expired requests. Bandaid also manages connections in user-space queues to avoid issues with kernel accept queues and delayed detection of closed connections. The service proxy enables weighted traffic management for canary deployments by assigning weights to queues belonging to a single upstream, allowing for dynamic configuration without restarts. It also supports logical isolation of critical and non-critical routes within the same host.
Advertisement
Sep 2017Optimizing web servers for high throughput and low latencyThis post details optimizations for web servers and proxies, focusing on hardware (CPU, NIC, Memory, Hard Drive), operating system low-level tunings (firmware, drivers, CPU governors, CPU affinity), and library/application-level tunings (specifically for Nginx). It discusses how to tune CPU performance for latency-sensitive vs. high-throughput tasks, NIC selection, memory requirements, and storage considerations. It also covers OS-level optimizations like using the 'performance' governor, disabling HT for latency-sensitive tasks, and utilizing tools like turbostat and cpupower. CPU affinity is discussed as a method to reduce latency by binding processes to cores, with considerations for potential performance degradation and monitoring with runqlat.