TechPrufer
Get started
Atlassian
Atlassian logoAtlassian · Feature trail

Online Index Migration and Shard Scaling for OpenSearch

5 posts · Oct 2018 – Jul 2026

What this trail covers

Atlassian has developed AOSC, an OpenSearch plugin enabling zero-downtime index migration and shard scaling. This system addresses challenges with modifying live indices by leveraging OpenSearch's internal capabilities like Retention Leases and Lucene Changes Snapshots to implement a CDC-like mechanism. It allows for online index reshaping and lays the groundwork for auto-scaling shards, significantly reducing operational burden and risk. Additionally, Atlassian is enhancing OpenSearch cluster management with Admission Control to improve resiliency by rate limiting incoming requests at the rest layer when a node is under high stress, preventing nodes from being overwhelmed and ensuring stable performance even under high demand. This includes tracking rolling average CPU and IO usage to reject search and bulk requests, returning HTTP 429 errors upon rejection. The system supports MONITOR and ENFORCED modes for admission control, with configurable limits for CPU/IO usage for indexing and search requests.

All Atlassian trails
  1. Online index migration and shard-scaling in OpenSearch with AOSCThis post details the creation and architecture of AOSC, an OpenSearch plugin that enables zero-downtime index migration and shard scaling. It explains how AOSC leverages OpenSearch's internal Retention Leases and Lucene Changes Snapshots to mimic Change Data Capture (CDC) for live index reshaping. The post discusses the challenges of existing OpenSearch APIs like _reindex, _split, and _shrink for live indices, and contrasts them with the application dual-write approach. It outlines the architecture of AOSC, including the use of Engine.Searcher for backfilling and index aliases for cutover, and discusses the trade-offs of a brief write block during the final cutover phase.
  2. Migrating the Jira Database Platform to AWS AuroraThis post details the migration of Jira's PostgreSQL database fleet to AWS Aurora. The migration strategy involved converting RDS instances to Aurora read replicas and then promoting them, with a critical challenge arising from excessive file counts on source instances that caused startup timeouts on new Aurora clusters. To overcome this, a 'draining' process was implemented to reduce file counts by migrating tenants to other instances before conversion, followed by a 'refill' process where converted Aurora clusters became targets for new migrations, creating a daisy-chain loop. This approach minimized migration-related infrastructure and maintained high throughput, averaging 38,000 daily migrations at peak. The process also involved careful management of source and destination concurrency to avoid impacting normal operations.
  3. Enhancing resiliency in opensearch clusters: An in-depth technical exploration of Admission ControlThis post details the implementation and benefits of OpenSearch's Admission Control feature for enhancing cluster resiliency. It explains the 'Why Admission Control?' and 'What is Admission Control?' aspects, including its role in rate-limiting requests based on resource utilization (CPU and IO). The technical breakdown covers the default settings, the three modes (DISABLED, MONITOR, ENFORCED), and provides detailed instructions on how to enable and configure admission control at different levels (transport, CPU/IO override, and specific limits for indexing/search). It also includes sample rejection responses and node stats API output, and discusses strategies for handling rejected requests, such as retries and Dead Letter Queues. The post demonstrates Atlassian's experience with admission control, showing improved search reliability during high indexing loads.
  4. Unlocking reliability with Splunk SmartStoreThis post details Atlassian's migration of its Splunk deployment from traditional EC2 instances with local NVMe storage to a SmartStore-enabled architecture utilizing AWS S3 for warm buckets. The migration addressed operational complexity and scaling challenges associated with growing log ingestion volumes. Key aspects include planning for minimal downtime and data loss, designing S3 gateway endpoints and bucket policies, configuring Splunk's `eviction_policy` to `lruk`, and automating the implementation using Ansible. The post highlights the successful decoupling of storage from compute, enabling reduced indexer node counts and improved scalability.
  5. Project RoombaThis post details the diagnosis and resolution of performance issues within Trello's Redshift database cluster, codenamed 'Project Roomba'. The core problems identified were inefficient data compression and sorting, leading to frequent long-running queries, query cancellations, and increased operational burden. The solution involved doubling the cluster size, replacing problematic tables with new versions utilizing high data compression encoding (which also re-sorted the data), and implementing ongoing sorting. This resulted in a 62% reduction in overall data size, a 65% reduction in the size of the 'atomic.events' table, and all events tables becoming 100% sorted. The improvements led to the elimination of 'killed' long-running queries, reduced the need for workaround code in data pipelines, fewer query alerts, and freed up data engineering time for other tasks.