Scaling Harvey’s Document Systems: Vault File Upload and Management

This post details significant architectural improvements to Harvey's Vault system, specifically focusing on scaling file upload and management for large document sets. Key technical contributions include: isolating upload operations into dedicated web workers, implementing separate state management for temporary vs. processing errors, building a robust file reconciliation system, and enabling intelligent parallel batching. The frontend dynamically adjusts parallelism based on file metadata and system load, while the backend consolidates queries into efficient bulk operations. For interface performance, the system migrated compute-intensive operations (sorting, filtering, aggregation) to the server-side, reducing browser memory consumption by 90% and improving initial load time by 80%. This involved developing server-driven search and file lists, cross-page file selection with lazy parent-mapping, event-driven context architecture, new bulk operation APIs, smart caching with predictive prefetching, and adaptive progress indicators.