Design a File-Processing API with Long-Running Jobs

You are tasked with designing an HTTP-based API and high-level backend architecture for processing files within a cloud storage service, similar to a simplified Dropbox. The system needs to perform CPU-intensive operations on files, such as virus scanning, OCR, or thumbnail generation.

Part 1: Simple Synchronous Processing API

Begin by designing a basic synchronous processing API:

• The client provides a list of file identifiers (e.g., file IDs or paths owned by the user).
• The backend must process each of these files and return the result for all files in a single response.
• Initially, assume that processing each file is relatively fast (e.g., under a second each) and the total request can reasonably complete within typical HTTP timeouts.

Your design should include:

• A clear REST-style API: define request and response formats for a synchronous endpoint like POST /processFiles.
• High-level backend components and data flow for handling this request.

Address the following in your design:

• How the client specifies which files to process.
• What information the response returns (per-file success/failure, output data, errors).
• Basic error handling (e.g., if some files fail).

Part 2: Handling Long-Running Requests (Follow-up)

Now, extend your design to robustly handle long-running processing. Assume that processing each file can take a significant amount of time, ranging from several seconds to several minutes, and the list of files may be extensive. A fully synchronous HTTP request will often exceed frontend or load balancer timeouts and provide a poor user experience.

New requirements for the extended design:

• The client can still submit a list of files to be processed.
• The request to submit a job should return quickly (within a few seconds) even if total processing will take minutes.
• The client must be able to:
  • Track the status of the processing job (e.g., pending, in-progress, completed, failed).
  • Obtain per-file results once processing is complete.
• The system should handle:
  • Server restarts and crashes.
  • Retries from clients (addressing idempotency concerns).
  • Scaling to many concurrent jobs.

What to Cover in Your Answer

Describe a high-level design that includes:

1. API Surface
   • Endpoints for:
     • Submitting a processing job for a list of files.
     • Checking job status.
     • Retrieving results (and whether status and results are combined or separate).
   • High-level request and response shapes.
2. Architecture and Components
   • How you will store jobs and their state (e.g., conceptual database schema).
   • How you will perform the actual file processing (e.g., background workers, message queues).
   • How work is distributed and scaled across multiple machines.
3. Long-Running Job Handling
   • How you avoid tying up HTTP connections for the duration of processing.
   • How the client can safely retry requests without creating duplicate jobs.
   • How to handle partial failures (some files succeed, others fail).
4. Reliability and Scalability Considerations
   • Handling failures and restarts: ensuring jobs are not lost and are eventually completed or marked failed.
   • Idempotency and deduplication strategies.
   • Basic performance considerations and bottlenecks.

You do not need to write actual code, but explain your design clearly enough that an experienced engineer could implement it from your description.