How we built Cloudflare's data platform and an AI agent on top of it

What this post added

This post details the creation of Cloudflare's internal data platform, Town Lake, and an AI data agent, Skipper. Town Lake is a data lakehouse architecture using Apache Trino as the query engine, R2 for object storage with Apache Iceberg for metadata, DataHub for metadata cataloging, Lifeguard for access control, Skimmer for PII detection using Workers AI, Transformer for ELT on Workflows, and a custom ingestion orchestrator. Skipper leverages Town Lake, Workers AI, Durable Objects, D1, R2, and Workflows to translate natural language questions into SQL queries, package results into dashboards, and assist with data transformation and access control.