Zone-Failure-Resilient OpenSearch

Uber has achieved Zone Failure Resilience (ZFR) for its OpenSearch deployments by integrating OpenSearch's native shard-allocation awareness features with its in-house isolation-group infrastructure. This approach ensures that OpenSearch clusters can withstand the complete loss of a zone without impacting core functions like querying and data ingestion. Key components include using isolation groups for balanced node distribution across failure domains and employing forced shard allocation awareness to prevent aggressive rebalancing during failures. The system can tolerate a full zone outage plus an additional node failure while still retaining quorum and preventing data loss, by running at least 3 shard copies and 5 cluster manager nodes with auto-shrink voting configuration.