Billions and billions (of logs): scaling AI Gateway with the Cloudflare Developer Platform

This post details the technical evolution of Cloudflare AI Gateway's log storage, migrating from a 30-minute retention period using D1 and a backend worker to indefinite storage. The solution involves storing request bodies in R2 and using Durable Objects (DOs) for log indexing and transactional storage. To overcome DO limitations (10 million logs per DO), logs are sharded by account ID and gateway name, raising the effective limit to 100 million logs per account. An Account Manager DO was introduced to track user entitlements, gateway limits, and log storage capacity, ensuring fair usage and system integrity. Gateways update the Account Manager periodically after every 1000 inference requests.