TechPrufer
Get started
Uber
Uber logoUber · Feature trail

Retry Storm Mitigation

3 posts · Apr 2026 – Sep 2026

What this trail covers

Uber has developed a context-aware mechanism in shared infrastructure to protect against retry storms. This system establishes error ownership by correlating inbound failures with outbound failures, allowing retries only when the error originates from the service itself. This prevents cascading failures caused by deep dependency chains and fan-out patterns, improving system reliability and user experience. This post details the large-scale automated dependency analysis across Uber's service mesh and introduces an intelligent load manager that detects overload from multiple signals to keep databases stable and fair under pressure, utilizing CoDel queues for controlled delay, a Scorecard engine for per-tenant fairness, and plug-in regulators for specific overload conditions like write bytes, partition keys, memory, and goroutines.

All Uber trails
  1. How Uber Protects Against Retry StormsThis post introduces a novel approach to mitigating retry storms by establishing 'error ownership'. It details a mechanism that correlates inbound service errors with outbound call failures, using a decision logic to determine if a service is the true 'owner' of an error. This allows for conditional retries, preventing upstream services from retrying when the error originates deeper in the dependency chain. The system leverages a Service Dependency Analysis Solution and introduces an 'x-uber-error-claim' header to propagate error ownership information. The post also discusses the challenges of coincidental errors and how the system can be adapted to avoid suppressing legitimate retries.
  2. Large-Scale Automated Dependency Analysis Across Uber's Service MeshThis post details the development of a large-scale automated dependency analysis system across Uber's service mesh. It introduces the concepts of fail-close and fail-open dependencies and explains how these are identified by correlating outbound failures with inbound requests. The system leverages yarpc middleware and unique request IDs to link outbound calls to their originating inbound requests, enabling the classification of dependencies. The post highlights the challenges of using distributed tracing at scale due to sampling rates and proposes a metric-based approach using on-host proxies (Muttley) and yarpc middleware to observe every failure and establish dependency correlations.
  3. How Uber Conquered Database Overload: The Journey from Static Rate-Limiting to Intelligent Load ManagementThis post details the development of an intelligent load manager for Uber's Docstore and Schemaless databases. It introduces a shift from quota-based rate limiting in the query engine to overload management within the stateful storage layer. Key contributions include the implementation of CoDel queues (Read, Write, Slow) with adaptive LIFO behavior for controlled delay, a Scorecard engine for per-tenant concurrency limits and fairness, and plug-in regulators (Write bytes, Partition key, Memory, Goroutines) to detect and mitigate specific overload conditions like I/O saturation, hot partition keys, and memory exhaustion. The post highlights improved availability and incident containment as key outcomes.