GitLab's data migration process for Advanced Search
Jun 1, 2021Dmitry Gruzd

Summary
Introduced a new, automated, background migration framework for Elasticsearch data, inspired by Rails DB migrations. This framework uses time-stamped migration files stored in Elasticsearch itself, with a background worker to execute them. It supports features like batched migrations, throttling, pausing indexing, and space requirement checks. This framework has been used to migrate issues, comments, and merge requests, and is crucial for the ongoing separation of document types into their own indices.
More from the Database Migration Debugging and Cross-Database Compatibility trail
- Seamlessly migrate from Jira to GitLab with Jira2Lab at scale
- We are upgrading the operating system on our Postgres database clusters
- Decomposing the GitLab backend database, Part 1: Designing and planning
- GitLab database decomposition: Final migration and results
- Decomposing the GitLab backend database, Part 3: Challenges and surprises