Indexing millions of HTTP requests using Durable Objects

What this post added

Introduced the R2 Log Retrieval API, built using Cloudflare Workers and Durable Objects, to index and query millions of customer logs stored in R2. This system uses Durable Objects for maintaining forward-indexes of RayIDs per log batch and the Streams API to process compressed log data without OOM errors. The indexing process involves decompressing, decoding, splitting, and collecting RayIDs, which are then stored in Durable Object storage. Log retrieval optimizes searches by leveraging the timestamp encoded in RayIDs and batch names to narrow down the search space, streaming relevant batches from R2 and filtering for matching records.