Problem Description

You are tasked with implementing a robust data ingestion function fetch_all_orders that retrieves order data from a paginated REST API, processes it, and writes it to a CSV stream.

API Endpoint Details

The API endpoint is GET /orders?page=<page_number>&limit=<records_per_page>. It returns JSON objects with the following structure:

{
  "page": <current_page_number>,
  "per_page": <records_per_page>,
  "total_pages": <total_number_of_pages>,
  "data": [
    // Array of order records
  ]
}

Function Signature

fetch_all_orders(base_url, limit, since=None, out_stream)

Requirements

Data Retrieval

• Complete Pagination: The function must retrieve all pages of data until total_pages is exhausted.
• since Parameter Support: If a since timestamp (UTC ISO 8601) is provided, only records with created_at >= since should be fetched and processed.

Data Processing

• Schema Flattening: Each retrieved order record must be flattened into a consistent schema:
  • id (string)
  • created_at (UTC ISO 8601)
  • amount_cents (integer)
  • customer_id (string)
  • customer_email (string)

Robustness & Performance

• Error Handling & Retries: Implement robust error handling for HTTP 429 (Too Many Requests) and 5xx (Server Error) responses. Retries should use exponential backoff with jitter, up to a configurable maximum number of retries.
• Rate Limiting: Ensure the ingestion process respects a strict rate limit of 10 requests per second to the API.

Output

• CSV Output: Write the flattened records to the provided out_stream in CSV format.
• Memory Efficiency: The implementation must avoid holding all records in memory simultaneously.
• Stable Ordering: Records in the CSV output must be sorted in stable order: ascending by created_at, then by id.

Deliverables

• Implementation: Provide the fetch_all_orders function.
• Unit Tests: Include unit tests covering:
  • Pagination boundary conditions (e.g., first page, last page, empty data).
  • Retry behavior for 429/5xx responses.
  • Correct schema parsing and flattening.
• Complexity Analysis: Discuss the time and space complexity of your solution.
• Adaptation Discussion: Explain how you would adapt your approach for a cursor-based pagination API, where the server returns a next_cursor token instead of total_pages.