Massively parallel Postgres backups
Jul 31, 2026Ben Dicken

Summary
This post details the engineering approach to achieving massively parallel Postgres backups for sharded databases (Neki). It describes spinning up dedicated backup nodes per shard, restoring from previous backups stored in object storage, and replaying WAL segments (hybrid approach using S3 and direct streaming from primary for recent changes) to catch up to the current state. The initial backup process using `pg_basebackup` is also outlined. The post quantifies the performance benefits of parallelism for large databases, demonstrating how it reduces backup times significantly and enables more frequent backups. It also explains how this backup system is leveraged for database resizing and node replacement operations.