Managing Elasticsearch Reindex at Scale: Performance, Reliability, and Observability

This post details the fourth iteration of Palantir's Elasticsearch reindex machinery, focusing on performance, reliability, and observability. Key contributions include the use of a shadow index with dual-writing and optimistic concurrency control for online reindexing, Elasticsearch index aliases for seamless switching, and a schema-level metadata table for orchestration. The system is designed to be cluster-agnostic, supporting both single-cluster and cross-cluster reindexing. It addresses requirements for schema-aware reconstruction, low manual action, automatic inconsistency detection, bounded resource usage, and online live reindexing. The post also discusses the evolution of the machine to handle increasing product features and usage, and storage infrastructure changes.