“Design a RAG system” is easy to answer badly: draw a vector database, wave at an LLM, say “we'll add reranking later.” The version that gets tested in ML system design interviews — and in production — adds one word: evaluation. Answer questions over a private corpus, keep documents fresh, cite sources, stay fast, avoid hallucinations, and measure every one of those claims.
The Interview Problem
Build natural-language Q&A over a private document corpus — internal docs, PDFs, knowledge-base articles — and walk through how you would evaluate each component. Four requirements do all the work:
Freshness
Frequent document updates must reach the index without downtime or stale answers.
Traceability
Every answer carries citations back to the exact source documents and pages.
Interactive latency
Low-latency responses while scaling from 100 to 1,000 requests per second.
Groundedness
Answers stay inside the retrieved context — and an evaluation plan proves it, stage by stage.
Phase 1: The Zero-Dollar MVP
Start with two pipelines on commodity hardware, all open source. Documents live in a Git-managed filesystem, get parsed with PyPDF2 / python-docx / markdown-it-py, split into 256–512-token chunks with overlap, embedded with all-MiniLM-L6-v2 (384 dimensions) on CPU, and indexed in-memory with Faiss. Queries embed with the same model, pull top-k (5–10) chunks, and a quantized Llama 2 7B generates via llama.cpp — answers post-processed to attach citations from chunk metadata.
Figure 1: The zero-dollar MVP — every box is open source and runs on CPU.
The Back-of-Envelope Math
Assume 1,000 internal documents averaging 1 MB, chunked at 256 tokens. This is where the design stops being hand-wavy:
Total Chunks
~976 chunks per document across 1,000 documents. Small enough for a single index, large enough that naive rebuilds hurt.
Embedding Storage
384 dims × 4 bytes = 1,536 B per chunk. The whole index fits in RAM on one commodity server.
Faiss Retrieval
Tens of milliseconds per query over ~1M vectors on CPU. Retrieval is not the problem.
LLM Inference on CPU
Quantized Llama 2 7B: 0.1–0.2 RPS per core. This is the fracture point.
corpus = 1,000 docs × 1 MB = 1 GB raw text chunks = 1,000 docs × ~976 chunks/doc = 976,000 chunks embeddings = 976,000 × 1,536 B = ~1.5 GB (fits in RAM ✓) retrieval = ~50 ms/query on CPU → fine at 100 RPS with parallelism generation = 5–10 s/query on CPU → 0.1–0.2 RPS per core cores @100RPS = 100 RPS ÷ 0.1 RPS/core = ~1,000 CPU cores ✗
Databricks' engineering team puts it bluntly: LLM inference can be 100× more expensive and 100× slower than traditional ML inference. Retrieval is cheap; generation is nearly everything.
Phase 2: Where It Fractures
Push the MVP toward 100–1,000 RPS and four things break, in order:
1. LLM inference throughput
CPU generation caps out three orders of magnitude below target. No amount of request queuing fixes a 5-second forward pass.
2. In-memory vector index
1.5 GB at 1K docs becomes 150 GB at 100K docs — past single-machine RAM, with ANN search contending for the same CPUs.
3. Ingestion freshness
Re-embedding 100 updated docs/day costs ~1.3 hours of CPU, and naive Faiss updates mean index rebuilds — downtime or stale answers.
4. Zero operational visibility
One machine, basic logs, no redundancy. Any crash takes the system down, and you cannot debug latency at 100 RPS from print statements.
Phase 3: The Evolutionary Forks
Each bottleneck has two escape paths. Interviewers love this framing because it shows you know why the managed service exists.
Path A: The No-Money Hack
- LLM: quantized GGUF models, a Redis answer cache, request batching, vLLM/TGI with paged attention on a consumer GPU.
- Index: hand-sharded Faiss across machines, disk-based ANN (Hnswlib), hybrid keyword fallback.
- Ingestion: off-peak cron batches, incremental re-indexing of changed chunks, Git hooks as triggers.
- Ops: self-hosted Prometheus + Grafana, structured JSON logs into ELK.
Path B: The Paid Solution
- LLM: managed APIs (GPT, Claude, Gemini) or vLLM/TensorRT-LLM on dedicated cloud GPUs.
- Index: managed vector DBs — Pinecone, Weaviate, Qdrant, Milvus. Uber runs 100K+ QPS at 20 ms latency on Milvus.
- Ingestion: S3/GCS event notifications → Lambda/Cloud Run workers, orchestrated by Step Functions or Airflow.
- Ops: Datadog/CloudWatch dashboards plus OpenTelemetry distributed tracing.
Evaluate Every Stage, Not Just the Answer
The differentiating requirement. A single end-to-end “does the answer look right?” check cannot tell you which stage failed. Instrument each one:
| Stage | Method | What it catches |
|---|---|---|
| chunking | Golden chunks + parse spot-checks | Mangled PDFs, chunk boundaries that split answers in half. |
| retrieval | Recall@k / Precision@k on a labeled set | Relevant chunks missing from top-k; noise crowding out signal. |
| generation | Human review + citation verification | Hallucinations; claims that trace to nothing that was retrieved. |
| end-to-end | User testing → online logs + feedback | Real query distribution drift; the continuous-improvement loop. |
The same layering guides model choice: Shopify found that fine-tuned smaller models or better prompting reach 80–90% of the quality of much larger models on specific tasks — a trade you can only make confidently when per-stage metrics exist.
Phase 4: The Final Boss — the Cloud Bill
At 1,000 RPS with managed LLM APIs, inference dominates everything else on the invoice:
1,000 RPS × 500 tokens/request = 500,000 tokens/s 500,000 tokens/s × $1.00 per 1M = $0.50/s $0.50/s × ~2.6M s/month ≈ $1.3M/month (inference alone) + managed vector DB ~$100s–$1,000s /month + ingestion compute, storage, network, observability ...
“The design review question is never ‘can we build RAG?’ — it's ‘who approved half a million tokens per second?’”
Permission-aware retrieval
Filter chunks by the caller's IAM entitlements before generation. A correctness issue, not a nice-to-have.
Long & multi-modal docs
Recursive/hierarchical chunking for documents past the context window; OCR and table extraction for images and scans.
Governance & feedback
GDPR/HIPAA over corpus and query logs; multi-turn query rewriting; a human feedback loop feeding the eval sets.
Takeaway
References & Further Reading
Practice This Question
This post is adapted from our interview question bank — the original comes with structured prep, a grounded four-phase model answer, and space for your own notes.