Online index migration and shard-scaling in OpenSearch with AOSC

This post details the creation and architecture of AOSC, an OpenSearch plugin that enables zero-downtime index migration and shard scaling. It explains how AOSC leverages OpenSearch's internal Retention Leases and Lucene Changes Snapshots to mimic Change Data Capture (CDC) for live index reshaping. The post discusses the challenges of existing OpenSearch APIs like _reindex, _split, and _shrink for live indices, and contrasts them with the application dual-write approach. It outlines the architecture of AOSC, including the use of Engine.Searcher for backfilling and index aliases for cutover, and discusses the trade-offs of a brief write block during the final cutover phase.