TechPrufer
Sign inGet started

Blogs›Uber›Zone-Failure-Resilient OpenSearch

Zone-Failure-Resilient OpenSearch

Zone-Failure-Resilient OpenSearch

1
posts
2026

Uber has achieved Zone Failure Resilience (ZFR) for its OpenSearch deployments by integrating OpenSearch's native shard allocation awareness with its in-house isolation-group infrastructure. This approach ensures that OpenSearch clusters can withstand the complete loss of a zone without impacting core functions like querying and data ingestion. Key components include using isolation groups for balanced node distribution across failure domains and employing forced shard allocation awareness to prevent disruptive rebalancing during zone outages. For cluster manager nodes, running 5 nodes with auto-shrink voting configuration further enhances resilience against multi-node failures.

2026

Zone-Failure-Resilient OpenSearch® at Uber

6/25/2026

This post details the implementation of Zone Failure Resilience (ZFR) for OpenSearch at Uber. It describes how Uber combines OpenSearch's shard allocation awareness with its isolation-group infrastructure to ensure cluster stability during zone failures. The post explains the concept of isolation groups, their role in balancing nodes across failure domains, and how shard allocation awareness distributes shard copies. It introduces 'forced shard allocation awareness' to prevent aggressive rebalancing during outages, thus protecting surviving zones. The resilience of data nodes (3 shard copies across 3 isolation groups) and cluster manager nodes (5 nodes with auto-shrink voting configuration) is also elaborated. The impact of using isolation groups over physical zone IDs for improved shard assignment and cluster health is highlighted.