TechPrufer
Get started
Cloudflare
Cloudflare logoCloudflare · Feature trail

Asynchronous Messaging & Queuing

21 posts · Mar 2018 – Oct 2026

What this trail covers

Cloudflare K2 is a new durable event streaming primitive on the Developer Platform, built on R2 object storage. It implements a partitioned, durable log for high-scale data movement, long-term retention, and fan-out consumption. K2 offers serverless operation, scales to vast quantities of data, and supports long-term retention. It differs from Queues by focusing on batch processing for high-scale data movement and fan-out consumption, whereas Queues are designed for tracking individual items of expensive or time-consuming work with message-level retries. K2's produce latency is approximately 1 second at the 99th percentile due to batching and writing to object storage. Future plans include higher write parallelism, message keys, push-based worker consumers, and Kafka client support.

All Cloudflare trails
  1. Announcing Cloudflare K2: serverless event streamsIntroduces Cloudflare K2, a new durable event streaming primitive built on R2 object storage. K2 implements a partitioned, durable log for high-scale data movement, long-term retention, and fan-out consumption. It addresses the challenge of decoupling producers and consumers by providing a service that absorbs writes and allows independent readers to consume at their own pace. K2's architecture relies on R2 for durable storage and strongly consistent APIs, simplifying the application layer. It accumulates writes in-memory on an edge service before writing them as segment files to R2, achieving ordering and strictly incrementing offsets using R2's atomic operations. The post highlights K2's approximately 1-second produce latency at the 99th percentile due to batching and object storage writes. It differentiates K2 from Cloudflare Queues (for individual work items with retries) and Basin Pipelines (for ingestion to object storage/Iceberg tables). The post details how to create streams, produce events via HTTP API or Worker binding, create subscriptions for read parallelism, and consume events with ack, nack, and extend lease operations. Beta limits include 10GB storage and 30 MB/s produce per stream. Future roadmap includes higher write parallelism, message keys, push-based worker consumers, and Kafka client support.
  2. How we built saga rollbacks for Cloudflare WorkflowsThis post introduces the implementation of the saga pattern with built-in rollback logic for Cloudflare Workflows. Developers can now define compensation logic for each `step.do()` call as an argument within the step itself. This simplifies the implementation of transactional workflows by ensuring that if a step fails, the system can automatically execute compensating actions in reverse order. The API design choices, including the use of an options object for rollback functions, are discussed, emphasizing the importance of idempotency and predictable execution order for rollbacks.
  3. Rearchitecting the Workflows control plane for the agentic eraIntroduced a new V2 control plane architecture for Workflows, featuring SousChef and Gatekeeper components to horizontally scale concurrency and instance creation. This addresses the bottleneck of the V1 single account-level Durable Object by distributing metadata and lifecycle management across subsets of instances per workflow (SousChef) and implementing a leasing system for concurrency slots (Gatekeeper). This enables significantly higher limits: 50,000 concurrent instances and 300 instances/second created per account. The migration strategy involved seamlessly transitioning existing customers and instances to the new architecture.
  4. Durable Objects in Dynamic Workers: Give each AI-generated app its own databaseIntroduces Durable Object Facets, a new feature that allows Dynamic Workers to dynamically load and instantiate Durable Object classes. This enables AI-generated code to manage persistent state within a secure sandbox, with a supervisor Durable Object handling logistics and providing each facet with its own SQLite database. The post provides a code example demonstrating how to implement this pattern.
  5. Building a better testing experience for Workflows, our durable execution engine for multi-step applicationsIntroduced new testing APIs (`introspectWorkflowInstance`, `introspectWorkflow`) for Cloudflare Workflows, enabling granular, isolated, and local testing with `vitest-pool-workers`. These APIs allow mocking of step results and events, disabling sleeps, and provide visibility into Workflow instance states, significantly improving the developer experience for testing complex, multi-step applications.
  7. Redesigning Workers KV for increased availability and faster performanceThis post details the complete redesign of Workers KV, a critical key-value store, following a major outage caused by reliance on a third-party cloud provider. The new architecture shifts to Cloudflare's own infrastructure, using a distributed database for small objects and R2 for larger ones, ensuring high availability, eliminating single points of failure, and improving performance. This represents a significant architectural shift for a core service.
  8. Cloudflare service outage June 12, 2025This post details a significant outage impacting Cloudflare Queues due to a dependency on Workers KV. The outage caused unavailability of message pushing and consuming operations. The root cause was identified as a failure in the underlying storage infrastructure for Workers KV, which Queues uses to map each Queue to its underlying Durable Objects. This incident highlights a critical dependency and a potential area for future resilience improvements in the Queues service.
  9. Let’s DO this: detecting Workers Builds errors across 1 million Durable ObjectsThis post details the development of an error detection system for Cloudflare Workers Builds. It describes the architecture of Workers Builds, which uses Durable Objects for build lifecycle management, and the challenges of identifying and categorizing build failures. The solution involves a new BuildErrorsQueue to process build logs, using static patterns and normalized log lines to detect and categorize errors. It also addresses the challenge of analyzing historical build logs stored across over a million Durable Objects by introducing a new approach to fetch and process these logs.
  10. How we simplified NCMEC reporting with Cloudflare WorkflowsThis post introduces Cloudflare Workflows as a solution to orchestrate complex asynchronous tasks that were previously managed with Cloudflare Queues. It highlights the limitations of a purely queue-based architecture for managing multi-step processes like NCMEC reporting, where manual state management, debugging across multiple queues, and unreliable retries were significant challenges. Cloudflare Workflows are presented as a more structured, observable, and retryable system that simplifies the definition and execution of these workflows, offering real-time visibility and easier troubleshooting.
  11. Making Super Slurper 5x faster with Workers, Durable Objects, and QueuesRearchitected Super Slurper using Cloudflare Workers, Durable Objects, and Queues to improve data transfer speeds by up to 5x. Addressed performance bottlenecks of the previous Kubernetes-based architecture by leveraging the scalability of Cloudflare's Developer Platform. Implemented a new processing layer with API Worker, List Queue Consumer, Transfer Queue Consumer Workers, and Lifecycle Queue Consumer. Introduced Durable Objects with sharding for managing billions of objects and used Hyperdrive for accessing legacy PostgreSQL data. Addressed challenges with duplicate messages in at-least-once delivery systems.
  12. Billions and billions (of logs): scaling AI Gateway with the Cloudflare Developer PlatformThis post details the technical evolution of AI Gateway's log storage, moving from a 30-minute retention period to indefinite storage for billions of logs. It describes the migration from D1 to R2 for request bodies, the use of D1 as a log index, and the subsequent adoption of Durable Objects with SQLite for persistent storage. Key technical challenges addressed include overcoming Durable Object storage limits by sharding logs by account ID and gateway name, and the introduction of an Account Manager Durable Object to manage user entitlements and track gateway usage, ensuring scalability and fair usage.
  13. Durable Objects aren't just durable, they're fast: a 10x speedup for Cloudflare QueuesThis post details the migration of Cloudflare Queues from a v1 architecture using a single Durable Object per queue to a v2 architecture that horizontally scales across multiple Durable Objects. The v1 architecture suffered from high latency due to centralized storage and limited throughput and consumer concurrency. The v2 architecture introduces Storage Shard Durable Objects distributed across regions for lower latency and multiple shards per region for increased throughput. Consumer Shard Durable Objects were introduced to overcome subrequest limits and increase concurrency. A Coordinator Durable Object manages a Shard Map stored in Workers KV to facilitate load balancing and shard discovery.
  14. Making state easy with D1 GA, Hyperdrive, Queues and Workers Analytics Engine updatesThis post announces the General Availability (GA) of Cloudflare Queues, marking it as production-ready. It introduces two major new features: pull-based consumers, which allow any HTTP-speaking client to pull and acknowledge messages, enabling consumers to run outside of Cloudflare Workers, and new message delivery controls including delayed messages. This significantly expands the flexibility and deployment options for asynchronous messaging.
  15. Cloudflare incident on October 30, 2023This post details a critical incident where a misconfiguration in a new deployment tool for Workers KV caused a 37-minute outage. The incident highlighted the dependency of many Cloudflare services on Workers KV and the challenges of performing rollbacks when critical internal tooling is also affected. The root cause was a bug in the deployment tool that incorrectly returned staging build GUIDs for production, leading to traffic being routed to an unauthorized version and resulting in HTTP 401 errors. The resolution involved a manual 'break glass' rollback to a known good build. Key follow-up actions include onboarding KV deployments to standardized Workers deployment models with automated impact detection and recovery, ensuring rollback processes work even when Cloudflare Access is down, and adding pre-checks to deployments.
  16. Debug Queues from the dash: send, list, and ack messagesIntroduced new debugging features for Cloudflare Queues within the dashboard: the ability to send messages directly to a queue, list queued messages with detailed metadata (body, timestamp, retries, source, type), and acknowledge (remove) messages. These features simplify testing and debugging of asynchronous message flows without requiring a consumer worker. The 'peek' functionality for listing messages does not affect message retry counts or queue state, unlike some other messaging systems. An HTTP Pull/Ack API is planned to leverage the same underlying infrastructure.
  17. Hardening Workers KVThis post details the hardening of Workers KV following a series of incidents. It addresses an observability gap in error reporting by improving how unhandled exceptions are surfaced in Workers. It also fixes a bug where a mutation to local state resulted in an unexpected mutation of global state, causing Workers KV to fail to re-enter its healthy state when data centers came back online. Additionally, a bug in the progressive release process for Workers KV was identified and fixed, which had inadvertently caused traffic to be black-holed due to an incorrectly defined pipeline configuration.
  18. Cloudflare Queues: messages at your speed with consumer concurrency and explicit acknowledgementThis post introduces two key enhancements to Cloudflare Queues: Consumer Concurrency and Explicit Acknowledgment. Consumer Concurrency automatically scales the number of consumers to match the rate of incoming messages, preventing backlogs. It autoscales based on queue size, message rate, and consumption success/failure, with a configurable `max_concurrency`. Explicit Acknowledgment allows consumers to individually acknowledge or retry messages within a batch, preventing reprocessing of successfully processed messages when a batch fails. This is achieved through new methods like `.ack()`, `.retry()`, `.ackAll()`, and `.retryAll()`. The post also mentions a quadrupling of message throughput, from 100 to over 400 messages per second, and outlines future plans for R2 as a direct consumer and an HTTP pull API.
  19. Announcing connect() — a new API for creating TCP sockets from Cloudflare WorkersIntroduced a new API in Cloudflare Workers for creating outbound TCP sockets, enabling direct connections to any TCP-based service. This API, named `connect()`, aims to be a standard across runtimes and supports opportunistic TLS via a `startTls()` method. It allows immediate writing to sockets before TLS handshake completion and simplifies TLS configuration by managing it via host infrastructure. The API is now integrated with the `pg` Node.js driver for PostgreSQL, allowing Workers to connect directly to databases.
  20. How we built an open-source SEO tool using Workers, D1, and QueuesThis post details the development of an open-source SEO tool called Prospector, which leverages Cloudflare Workers, D1, and Queues. It demonstrates how these services can be integrated to build a functional application. Prospector uses Workers as the user-facing API, D1 for data storage and retrieval, and Queues for handling URL fetching and notification processes. The post also highlights the use of Hono framework for API development and improved TypeScript bindings for Workers.
  21. Intelligent, automatic restarts for unhealthy Kafka consumersIntroduced an intelligent health check for Kafka consumers that monitors message ingestion and offset commits, rather than just broker connectivity. This addresses issues where consumers appear healthy but are not processing messages, especially during rebalances. The solution involves tracking committed offsets per partition and failing the health check if the committed offset has not changed since the last check or if it equals the current offset. A fix was implemented to handle rebalances by rebuilding the in-memory offset map to only include currently assigned partitions, preventing cascading failures.
  22. Build applications of any size on Cloudflare with the Queues open betaThis post introduces the open beta of Cloudflare Queues, a serverless message queuing service integrated with Cloudflare Workers. It demonstrates how to enroll in the beta, create queues via the dashboard and Wrangler CLI, and build a practical log aggregator application. The application uses a Worker with `fetch` and `queue` handlers to receive log lines, send them to a queue, and then batch and write them to an R2 bucket. A real-world example from UUID.rocks is also presented, showcasing how Queues can buffer R2 writes to reduce costs and improve efficiency. The post also briefly touches on the internal architecture of Queues, built using Workers and Durable Objects.
  23. Cloudflare Queues: globally distributed queues without the egress feesIntroduced Cloudflare Queues, a global message queuing service for Cloudflare Workers. It offers at-least-once message delivery, supports batching, and has no bandwidth egress fees. The service integrates with Workers via producer and consumer APIs, using a `queue` event handler for message processing. Configuration options include batch size, retries, delivery wait time, and dead-letter queues. It supports custom routing logic via JavaScript and future integrations with R2. Pricing is $0.40 per million operations (64KB chunks).
  24. Building a scheduling system with Workers and Durable ObjectsIntroduced a new scheduling system using Cloudflare Workers and Durable Objects. Each scheduled request is managed by a unique Durable Object that stores request metadata and uses Durable Object Alarms to trigger HTTP requests at specified times or intervals. The system supports creating, overriding, and managing scheduled requests via a POST API, with Durable Object IDs used for identification and management.
  25. Using Apache Kafka to process 1 trillion inter-service messagesThis post details Cloudflare's extensive use of Apache Kafka for inter-service communication, processing over a trillion messages. It introduces internal tooling like Messagebus-Client (a Go library wrapping Sarama) and a connector framework based on Kafka Connect for data pipelines. The post emphasizes strict schema enforcement using Protobuf and a schema registry, and details observability tooling including automated metrics and alerts for Kafka topics. A practical example of the Alert Notification System (ANS) is provided, showcasing how teams can easily integrate new alert types by emitting messages to Kafka.
  26. Squeezing the firehose: getting the most from Kafka compressionThis post details Cloudflare's experience and performance testing with Kafka compression, specifically focusing on the improvements introduced in Kafka 0.11.0. It covers the evolution of Kafka's compression mechanism from individual message compression to batch compression, the trade-offs between CPU and disk/network usage, and the bottlenecks encountered at Cloudflare (disk I/O, network saturation). The post presents performance testing results for different compression codecs (gzip, lz4, snappy) on various topics, including HTTP request logs encoded with capnp, and analyzes the CPU usage of the Snappy compression algorithm in Go producers. It also includes a thread stacktrace from a Kafka broker highlighting the Snappy compression process.