Billions and billions (of logs): scaling AI Gateway with the Cloudflare Developer Platform

What this post added

This post details the evolution of AI Gateway's logging capabilities, addressing the critical challenge of storing billions of logs indefinitely. It describes the transition from a 30-minute log retention limit to persistent storage by migrating request bodies to R2 and utilizing Durable Objects with SQLite for indexed logs. The introduction of sharding by account ID and gateway name, along with an Account Manager Durable Object for entitlement and usage tracking, significantly scales log storage capacity and management. This enables advanced features like AI evaluations and Human-in-the-Loop feedback by providing persistent, searchable historical data.