We are upgrading the operating system on our Postgres database clusters

What this post added

This post details the upgrade of the operating system on GitLab.com's Postgres database clusters from Ubuntu 16.04 to Ubuntu 20.04. The primary technical challenge addressed is the corruption of Postgres indexes caused by an updated glibc version in Ubuntu 18.10 and later. The chosen approach involves using streaming replication to new Ubuntu 20.04 standby clusters, followed by a planned downtime window to promote the new clusters, reindex affected indexes (specifically B-tree, GiST, and Hash, with a concurrent strategy for GIN indexes), and update DNS. Significant performance improvements were made to the `amcheck` function for faster identification of corrupt B-tree indexes and to the reindexing process itself, reducing the overall downtime required.