Garbage Day: How Pinecone Safely Deletes Billions of Objects at Scale

What this post added

This post introduces 'Janitor', a new system designed to safely delete billions of objects at scale in Pinecone's immutable blob storage. It details the challenges of managing stale and orphaned objects due to immutability and propagation lag. The system is broken down into three modes: 'Normal' for everyday superseded files, 'Orphan' for files from crashed processes, and 'Customer Deletion' for user-initiated data removal with a 30-day recovery window. The core protocol involves 'Identify' (candidate generation), 'Verify' (rechecking reachability against fresh system state), and 'Execute' (safe, idempotent deletion). It also highlights the use of property-based testing with a mock clock to simulate and test complex, time-dependent behaviors over long periods.