How Rapidflare built a million+ document ingestion pipeline for Agents on Temporal

This post details Rapidflare's implementation of a million+ document ingestion pipeline using Temporal. Key technical contributions include: 1. Using Temporal Activity Heartbeats for long-running crawlers, sending meaningful progress updates instead of generic heartbeats. 2. Setting `maximum_attempts=1` on crawler Activities, relying on internal crawler retry logic to avoid discarding progress. 3. Utilizing Cloud Storage as a data bus for large artifacts (beyond Temporal payload limits) to enable state sharing between Activities. 4. Implementing a sliding window mechanism with Child Workflows and `workflow.wait_condition` to control concurrency and prevent overwhelming downstream rate-limited services. 5. Employing Continue-as-New at page boundaries to manage large Workflow state and keep history limits in check. 6. Using deterministic Child Workflow IDs for predictable conflict resolution. 7. Designing a source-agnostic pipeline with three Temporal Workers on separate Task Queues for isolation (Ingestion, Enrichment, Status sync).