Zone-Failure-Resilient OpenSearch® at Uber

What this post added

This post details the implementation of Zone Failure Resilience (ZFR) for OpenSearch at Uber. It describes how Uber combines OpenSearch's shard allocation awareness with its isolation-group infrastructure to ensure cluster stability during zone failures. The post explains the concept of isolation groups, their role in balancing nodes across failure domains, and how shard allocation awareness distributes shard copies. It introduces 'forced shard allocation awareness' to prevent aggressive rebalancing during outages, thus protecting surviving zones. The resilience of data nodes (3 shard copies across 3 isolation groups) and cluster manager nodes (5 nodes with auto-shrink voting configuration) is also elaborated. The impact of using isolation groups over physical zone IDs for improved shard assignment and cluster health is highlighted.