TechPrufer
Get started
Pinecone
Pinecone logoPinecone · Feature trail

Object Lifecycle Management

1 post · Mar 2023 – Mar 2026

What this trail covers

Pinecone has implemented 'Janitor', a robust system for managing object lifecycle and safely deleting billions of objects at scale. This system addresses the challenges of immutable storage by identifying and removing stale and orphaned objects through distinct 'Normal', 'Orphan', and 'Customer Deletion' modes. Janitor employs a core protocol of 'Identify -> Verify -> Execute' to ensure safe and auditable deletion, with advanced property-based testing using a mock clock to validate complex, time-dependent behaviors.

All Pinecone trails
  1. Garbage Day: How Pinecone Safely Deletes Billions of Objects at ScaleThis post details the design and implementation of 'Janitor', a system for safely deleting billions of objects from immutable blob storage at Pinecone. It introduces three deletion modes (Normal, Orphan, Customer Deletion) and a core protocol (Identify -> Verify -> Execute) to manage stale and orphaned data. The system uses property-based testing with a mock clock to simulate long-term behaviors and ensure reliability. The primary technical contribution is the robust, auditable, and safe deletion mechanism for large-scale immutable storage.
  2. March 1st Partial Database OutageThis post details a critical incident where a bug in an inactivity-checking script, combined with an inappropriate deployment process, led to the accidental deletion of 515 free-tier indexes. The root cause was a faulty SQL query and the use of a lighter deployment process for a data-modifying script. The incident response involved scaling down operators, engaging cloud provider support for recovery, and a multi-day restoration process. Lessons learned emphasize the need for a rigorous database culture, specific design considerations for the free plan, improved testing and shipping processes for critical features, and the prioritization of soft deletions and data exports.