How Rapidflare built a million+ document ingestion pipeline for Agents on Temporal

What this post added

This post details Rapidflare's implementation of a million+ document ingestion pipeline using Temporal. Key technical contributions include: a sliding window mechanism for managing a large number of Child Workflows to control concurrency and maintain worker utilization; the use of Continue-as-New to manage large Workflow histories by checkpointing state at page boundaries; the implementation of Activity Heartbeats with milestone reporting for robust progress tracking during long-running crawl Activities; the use of Cloud Storage as a data bus for staging large document artifacts, overcoming Temporal's payload limits and enabling shared state across distributed workers; deterministic Child Workflow IDs to prevent duplicate processing; and the `ParentClosePolicy.ABANDON` to ensure children survive parent restarts. The post also highlights the use of separate Task Queues for different processing stages (Ingestion, Enrichment, Status sync) to provide isolation and prevent starvation.