From 0 to 20 billion - How We Built Crawler Hints

This post details the engineering challenges and solutions for building Cloudflare's Crawler Hints product. It describes the system architecture, including the use of Kafka for data pipelines, Redis as a distributed buffer for aggregation and deduplication, and a dispatcher service for sending batches of URLs to search partners via the IndexNow API. The post highlights the importance of customer opt-in, monitoring, and system resilience for rolling out such a large-scale product.