Managing Elasticsearch Reindex at Scale: Performance, Reliability, and Observability

This post details the fourth iteration of Palantir's Elasticsearch reindex machinery, focusing on performance, reliability, and observability. Key contributions include the use of a shadow index with dual-writing and optimistic concurrency control for live reindexing, Elasticsearch index aliases for seamless switching, and a schema-level metadata table for orchestration and autonomous detection of reindex needs. The system is designed to be cluster-agnostic, supporting both single-cluster and cross-cluster reindexing. This iteration also introduces mechanisms for bounded resource usage and online reindexing without service downtime.