Scaling Document Processing Across Harvey

This post details the infrastructure changes made to scale Harvey's document processing capabilities from handling under a million documents weekly to tens of millions. The core contribution is the architectural split of the document processing pipeline into three distinct, independently scalable systems: extraction, chunking/embedding, and indexing. This was enabled by migrating to a new Job Framework for durable and observable background work, introducing the Unified Document Format (UDF) to manage intermediate document representations, and optimizing the indexing hot path with a live vector database migration and the adoption of Arrow IPC for serialization/deserialization. The system now implements backpressure mechanisms to handle downstream rate limits and failures gracefully.