Rearchitecting the Workflows control plane for the agentic era

What this post added

Introduced a new V2 control plane architecture for Workflows, featuring SousChef and Gatekeeper components to horizontally scale concurrency and instance creation. This addresses the bottleneck of the V1 single account-level Durable Object by distributing metadata and lifecycle management across subsets of instances per workflow (SousChef) and implementing a leasing system for concurrency slots (Gatekeeper). This enables significantly higher limits: 50,000 concurrent instances and 300 instances/second created per account. The migration strategy involved seamlessly transitioning existing customers and instances to the new architecture.