Building a New File Ingestion System to Scale Firm Knowledge

This post details the development of a new file ingestion system to scale firm knowledge by enabling folder uploads and one-way sync from document management systems. It introduces an asynchronous workflow architecture using Temporal for orchestration, focusing on resilience through granular activity design for crawling and isolated file downloads. A custom in-house rate limiter built on Redis was implemented to manage external API quotas. A diffing algorithm using external IDs, hash-based comparison, and timestamp fallbacks ensures correctness for sync operations. Performance optimizations include aggressive batching, connection pooling, parallelization, using blob storage for intermediate states, careful task queue design, and immediate cleanup of temporary data.