Needle in a haystack: efficient storage of billions of photos

This post details the transition from a metadata-heavy NFS-based photo storage infrastructure to a new system called Haystack. Haystack merges the photo serving and storage tiers into a single physical tier, utilizing an HTTP-based photo server that stores photos in a generic object store. The primary goal was to eliminate unnecessary metadata overhead for read operations. Haystack is built on commodity storage blades with RAID-6, uses XFS as an extent-based filesystem, and implements an append-only object store with an index file for efficient lookups. The Photo Store server maintains an in-memory index of photo offsets for rapid retrieval. The post also describes Haystack's write, read, and delete operations, as well as the compaction process for reclaiming space. This new infrastructure significantly improved performance by reducing I/O operations per photo.