Zone-Failure-Resilient OpenSearch® at Uber

This post details the implementation of Zone Failure Resilience (ZFR) for OpenSearch at Uber. It explains how Uber combines OpenSearch's shard allocation awareness with its internal isolation group infrastructure to ensure resilience against zone failures. Key technical contributions include: 1. Using isolation groups as an abstraction layer to balance nodes across failure domains, ensuring even distribution of shards. 2. Leveraging forced shard allocation awareness to prevent disruptive rebalancing during zone failures, trading immediate full replication for cluster stability. 3. Enhancing cluster manager node resiliency by running 5 nodes and enabling `cluster.auto_shrink_voting_configuration` to maintain quorum during failures. The post also highlights the impact of using isolation groups over physical zone IDs for improved shard assignment and cluster health.