Everything in its write place: Cloud storage abstraction with Object Store

This post details the design and implementation of Object Store, an abstraction layer over multiple cloud storage backends (S3, HDFS, and internally Magic Pocket). It addresses cost inefficiencies of using S3 by batching small PUT requests into larger blobs, reducing API request costs. It also introduces per-object encryption using AES-256 with a unique block encryption key (BEK) for each object, which is itself encrypted by a global key encryption key (KEK) for secure key rotation. For deletions, it implements crypto-shredding, where encrypting the BEK allows for effective deletion by simply removing the encrypted BEK from metadata, without requiring immediate data purging from the underlying store.