Asynchronous Messaging & Queuing

4/15/2026

Introduced a new V2 control plane architecture for Workflows, featuring SousChef and Gatekeeper components to horizontally scale concurrency and instance creation. This addresses the bottleneck of the V1 single account-level Durable Object by distributing metadata and lifecycle management across subsets of instances per workflow (SousChef) and implementing a leasing system for concurrency slots (Gatekeeper). This enables significantly higher limits: 50,000 concurrent instances and 300 instances/second created per account. The migration strategy involved seamlessly transitioning existing customers and instances to the new architecture.

8/8/2025

This post details the complete redesign of Workers KV, a critical key-value store, following a major outage caused by reliance on a third-party cloud provider. The new architecture shifts to Cloudflare's own infrastructure, using a distributed database for small objects and R2 for larger ones, ensuring high availability, eliminating single points of failure, and improving performance. This represents a significant architectural shift for a core service.

6/13/2025

This post details a significant outage impacting Cloudflare Queues due to a dependency on Workers KV. The outage caused unavailability of message pushing and consuming operations. The root cause was identified as a failure in the underlying storage infrastructure for Workers KV, which Queues uses to map each Queue to its underlying Durable Objects. This incident highlights a critical dependency and a potential area for future resilience improvements in the Queues service.

5/29/2025

This post details the development of an error detection system for Cloudflare Workers Builds. It describes the architecture of Workers Builds, which uses Durable Objects for build lifecycle management, and the challenges of identifying and categorizing build failures. The solution involves a new BuildErrorsQueue to process build logs, using static patterns and normalized log lines to detect and categorize errors. It also addresses the challenge of analyzing historical build logs stored across over a million Durable Objects by introducing a new approach to fetch and process these logs.

4/11/2025

This post introduces Cloudflare Workflows as a solution to orchestrate complex asynchronous tasks that were previously managed with Cloudflare Queues. It highlights the limitations of a purely queue-based architecture for managing multi-step processes like NCMEC reporting, where manual state management, debugging across multiple queues, and unreliable retries were significant challenges. Cloudflare Workflows are presented as a more structured, observable, and retryable system that simplifies the definition and execution of these workflows, offering real-time visibility and easier troubleshooting.

4/10/2025

Rearchitected Super Slurper using Cloudflare Workers, Durable Objects, and Queues to improve data transfer speeds by up to 5x. Addressed performance bottlenecks of the previous Kubernetes-based architecture by leveraging the scalability of Cloudflare's Developer Platform. Implemented a new processing layer with API Worker, List Queue Consumer, Transfer Queue Consumer Workers, and Lifecycle Queue Consumer. Introduced Durable Objects with sharding for managing billions of objects and used Hyperdrive for accessing legacy PostgreSQL data. Addressed challenges with duplicate messages in at-least-once delivery systems.

10/24/2024

This post details the technical evolution of AI Gateway's log storage, moving from a 30-minute retention period to indefinite storage for billions of logs. It describes the migration from D1 to R2 for request bodies, the use of D1 as a log index, and the subsequent adoption of Durable Objects with SQLite for persistent storage. Key technical challenges addressed include overcoming Durable Object storage limits by sharding logs by account ID and gateway name, and the introduction of an Account Manager Durable Object to manage user entitlements and track gateway usage, ensuring scalability and fair usage.

10/24/2024

Introduced a v2 architecture for Cloudflare Queues that horizontally scales across multiple Durable Objects to implement each high-performance Queue. This involves distributing Storage Shard Durable Objects in all available regions for lower latency and using multiple shards within each region to scale throughput. A new Consumer Shard Durable Object class was created to scale horizontally, enabling more concurrent invocations of user's `queue()` handlers. A Coordinator Durable Object was introduced as the 'Control Plane' to manage the individual Durable Objects in the system and maintain a Shard Map in Workers KV.

4/1/2024

This post announces the General Availability (GA) of Cloudflare Queues, marking it as production-ready. It introduces two major new features: pull-based consumers, which allow any HTTP-speaking client to pull and acknowledge messages, enabling consumers to run outside of Cloudflare Workers, and new message delivery controls including delayed messages. This significantly expands the flexibility and deployment options for asynchronous messaging.

11/1/2023

This post details a critical incident where a misconfiguration in a new deployment tool for Workers KV caused a 37-minute outage. The incident highlighted the dependency of many Cloudflare services on Workers KV and the challenges of performing rollbacks when critical internal tooling is also affected. The root cause was a bug in the deployment tool that incorrectly returned staging build GUIDs for production, leading to traffic being routed to an unauthorized version and resulting in HTTP 401 errors. The resolution involved a manual 'break glass' rollback to a known good build. Key follow-up actions include onboarding KV deployments to standardized Workers deployment models with automated impact detection and recovery, ensuring rollback processes work even when Cloudflare Access is down, and adding pre-checks to deployments.

8/11/2023

Introduced new debugging features for Cloudflare Queues within the Cloudflare dashboard: the ability to send messages directly, list queued messages with detailed metadata (timestamp, retries, producer source, type), and acknowledge (remove) messages. This enhances transparency and simplifies the debugging of asynchronous message flows without requiring a consumer worker or interfering with existing consumers.

8/2/2023

This post details the hardening of Workers KV following a series of incidents. It addresses an observability gap in error reporting by improving how unhandled exceptions are surfaced in Workers. It also fixes a bug where a mutation to local state resulted in an unexpected mutation of global state, causing Workers KV to fail to re-enter its healthy state when data centers came back online. Additionally, a bug in the progressive release process for Workers KV was identified and fixed, which had inadvertently caused traffic to be black-holed due to an incorrectly defined pipeline configuration.

5/19/2023

Introduced consumer concurrency for automatic scaling of message consumers based on queue load, and explicit acknowledgment methods (.ack(), .retry(), .ackAll(), .retryAll()) to allow individual message processing control, preventing reprocessing of successfully processed messages within a batch. Also noted a quadrupling of message throughput from 100 to over 400 messages per second.

3/2/2023

This post details the development of an open-source SEO tool called Prospector, which leverages Cloudflare Workers, D1, and Queues. It demonstrates how these services can be integrated to build a functional application. Prospector uses Workers as the user-facing API, D1 for data storage and retrieval, and Queues for handling URL fetching and notification processes. The post also highlights the use of Hono framework for API development and improved TypeScript bindings for Workers.

1/24/2023

Introduced intelligent health checks for Kafka consumers deployed on Kubernetes. These checks monitor message ingestion by comparing current and committed offsets, and ensure that committed offsets are actively changing. Implemented a solution to handle rebalances by rebuilding the in-memory offset map based on partition reassignment signals from the Sarama library, preventing cascading failures.

11/14/2022

This post announces the open beta of Cloudflare Queues, a new message queuing service integrated with Cloudflare Workers. It provides a practical example of building a log sink using Queues and R2, demonstrating how to send messages to a queue and consume them in batches for storage. It also briefly touches on how Queues were built using other Cloudflare services like Workers and Durable Objects.

9/27/2022

This post introduces Cloudflare Queues, a new global message queuing service for Cloudflare Workers. It details the core concepts of message queues, the producer/consumer model using Workers, message encoding via structuredClone(), batching, retries, and dead-letter queues. It also outlines configuration options in `wrangler.toml`, potential use cases, pricing ($0.40 per million operations), and the beta waitlist. It highlights the absence of egress fees, contrasting with AWS SQS and Google Pub/Sub.

8/5/2022

Introduced a new scheduling system using Cloudflare Workers and Durable Objects. Each scheduled request is managed by a unique Durable Object that stores request metadata and uses Durable Object Alarms to trigger HTTP requests at specified times or intervals. The system supports creating, overriding, and managing scheduled requests via a POST API, with Durable Object IDs used for identification and management.

7/19/2022

This post details Cloudflare's extensive use of Apache Kafka for inter-service communication, processing over a trillion messages. It introduces internal tooling like Messagebus-Client (a Go library wrapping Sarama) and a connector framework based on Kafka Connect for easier data pipeline creation. The post also highlights the importance of strict protobuf schemas managed via Messagebus Schema for forward/backward compatibility and code generation, and discusses observability strategies using Prometheus, Grafana, and automated alerts for high lag. A practical example of the Alert Notification System (ANS) is provided to illustrate how these tools facilitate new alert types.

3/5/2018

This post details Cloudflare's experience and performance testing with Kafka compression, specifically focusing on the improvements introduced in Kafka 0.11.0. It covers the evolution of Kafka's compression mechanism from individual message compression to batch compression, the trade-offs between CPU and disk/network usage, and the bottlenecks encountered at Cloudflare (disk I/O, network saturation). The post presents performance testing results for different compression codecs (gzip, lz4, snappy) on various topics, including HTTP request logs encoded with capnp, and analyzes the CPU usage of the Snappy compression algorithm in Go producers. It also includes a thread stacktrace from a Kafka broker highlighting the Snappy compression process.