Design a PDF-to-Markdown Inference API
Problem Description
Design an inference service responsible for converting PDF files into Markdown format. You can assume the existence of the following pre-built, specialized components:
- A CPU-intensive function that efficiently splits a PDF document into individual pages and converts each page into a NumPy array.
- A GPU-intensive Optical Character Recognition (OCR) engine.
- A memory-intensive post-processing step that transforms OCR outputs into Markdown or assembles the final page results into a complete document.
The design must address two distinct operational scenarios:
Scenario 1: Synchronous Conversion for Large Documents
Design an API for the synchronous conversion of a single, very large document (e.g., a 1000-page PDF). The primary objective is to deliver the full converted Markdown output to the user as quickly as possible.
Scenario 2: Asynchronous Conversion for Concurrent Requests
Design an API to handle numerous concurrent conversion requests asynchronously. In this scenario, clients expect to receive their conversion results at a later time, prioritizing overall system throughput and responsiveness.
Your explanation should cover the following architectural and operational aspects:
- API contract
- Page-level parallelism
- CPU and GPU scheduling
- Batching strategies
- Result ordering
- Intermediate storage mechanisms
- Fault tolerance
- Backpressure mechanisms
- Strategies for system scaling
AI interview prep
Core dilemma
Balancing the optimization of heterogeneous, resource-intensive workloads (CPU, GPU, memory) for both low-latency single-job processing and high-throughput multi-job processing, while ensuring system reliability, fault tolerance, and cost-effective scalability.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design a PDF-to-Markdown Inference API
Problem Description
Design an inference service responsible for converting PDF files into Markdown format. You can assume the existence of the following pre-built, specialized components:
- A CPU-intensive function that efficiently splits a PDF document into individual pages and converts each page into a NumPy array.
- A GPU-intensive Optical Character Recognition (OCR) engine.
- A memory-intensive post-processing step that transforms OCR outputs into Markdown or assembles the final page results into a complete document.
The design must address two distinct operational scenarios:
Scenario 1: Synchronous Conversion for Large Documents
Design an API for the synchronous conversion of a single, very large document (e.g., a 1000-page PDF). The primary objective is to deliver the full converted Markdown output to the user as quickly as possible.
Scenario 2: Asynchronous Conversion for Concurrent Requests
Design an API to handle numerous concurrent conversion requests asynchronously. In this scenario, clients expect to receive their conversion results at a later time, prioritizing overall system throughput and responsiveness.
Your explanation should cover the following architectural and operational aspects:
- API contract
- Page-level parallelism
- CPU and GPU scheduling
- Batching strategies
- Result ordering
- Intermediate storage mechanisms
- Fault tolerance
- Backpressure mechanisms
- Strategies for system scaling