Needle in a haystack: efficient storage of billions of photos

What this post added

This post details the transition from an NFS-based photo storage infrastructure to a new system called Haystack. The primary technical contribution is the design and implementation of Haystack, an object store that addresses the metadata overhead of traditional file systems. Haystack uses an append-only log structure with a separate index file to store photos, minimizing I/O operations per read. It also introduces a Photo Store Server that maintains an in-memory index for fast lookups and handles photo uploads, modifications, and deletions by interacting with Haystack. The post also discusses the underlying storage (commodity blades with RAID-6) and filesystem (XFS) choices, and the strategies for handling deleted needles and compaction to reclaim space.