Most RAG designs are drawn as if the corpus holds still. It doesn't. The interview prompt — question answering over a continuously updated internal corpus, with citations, low latency, and PDFs/HTML/wiki pages all mixed together — is really an optimization problem: five components that each break at a different scale, and a budget that decides which fork you take at each one.
What the Prompt Actually Asks
Design the end-to-end pipeline — ingestion, chunking, embeddings, index, retriever, reranker, generator — then defend three claims with numbers:
- Freshness: new and updated documents flow in continuously — no batch rebuilds, no stale answers.
- Quality: high answer quality with citations, which means reducing hallucinations by construction, not by prompt-begging.
- Latency: interactive use — and an evaluation plan (offline + online) that catches drift before users do.
Phase 1: The Zero-Dollar MVP
Bootstrap everything on one machine with open source. A manual script reads files with pypdf and BeautifulSoup, chunks naively at 500 characters with 50 overlap, embeds with all-MiniLM-L6-v2 (384-d) on CPU, and stores vectors in an in-memory Faiss index with chunk metadata in SQLite. Queries run k-NN for the top 5 chunks — no reranker — and a quantized 7B model (Llama-2 or Mistral via llama.cpp) writes the answer.
Figure 1: The zero-dollar MVP — one machine, no reranker, eventual everything.
corpus (1 GB text, 500-char chunks) ≈ 1,000,000 vectors
index = 1M × 384 dims × 4 B (fp32) = 1.5 GB → fits in RAM ✓
query embed (MiniLM, CPU) = 10–50 ms ✓
faiss k-NN over 1M vectors = tens of ms ✓
7B LLM inference: CPU → seconds/query
RTX 3060 → 1–2 s/query
100 RPS × 1–2 s = 100–200 s of compute per wall-clock second ✗
honest capacity: 1–5 RPS, single-user latency onlyEvery stage upstream of the LLM clears the bar. Generation alone caps the MVP at 1–5 RPS — the 100 RPS target is aspirational the day the system ships.
Phase 2: Five Fracture Points
Grow the corpus and the traffic, and the MVP fractures in five distinct places — each with its own arithmetic. Memorize the numbers, not the slogans:
Manual ingestion per day
1,000 new or updated documents daily × 5 minutes of hand-holding each = ~83 hours of human work per day. The cron-and-pray script dies first.
CPU-bound embedding backlog
200,000 new chunks/day × 50 ms per MiniLM embedding = 10,000 seconds of continuous CPU — competing with query-time embedding for the same cores.
RAM just for the index
10M chunks × 384 dims × 4 bytes. In-memory Faiss on one node runs out of RAM before it runs out of documents — and it is a single point of failure.
Precision at k=5
Naive fixed-size chunks + raw k-NN means 2 of 5 retrieved chunks are noise. No reranker → noisy context → hallucinations. Quality, not RPS, breaks here.
LLM instances for 100 RPS
A 7B model at 2 s/inference on CPU needs 200 parallel instances to hold 100 RPS. Impossible on one machine, and users still wait >5 s per answer.
Notice that only two of the five are throughput problems. Ingestion and embedding are freshness problems, and retrieval precision is a quality problem — the answer feels slow because it's wrong, and the user's time-to-resolution balloons even while your p99 looks healthy.
Phase 3: The Evolutionary Forks
Every bottleneck forks into a no-money hack and a paid solution. The optimization skill is knowing which fork to take per component — not one answer for the whole system.
| Bottleneck | No-money hack | Paid solution |
|---|---|---|
| ingestion | unstructured.io OSS + pypdf + BeautifulSoup on a cron; inotify triggers | S3 events → Lambda → AWS Textract / Unstructured managed API |
| embeddings | Quantized MiniLM, batch queue on a refurbished GPU box | OpenAI text-embedding-3-large (~$0.20/M tokens) or Cohere Embed v3 |
| vector index | Disk-based ANN (Annoy / Hnswlib) + hand-rolled shards + Redis metadata | Managed vector DB — Milvus: 1B vectors, 99% recall at 10 ms |
| retrieval | Semantic chunking + self-hosted BM25 hybrid + recency heuristics | Cohere Rerank over the top 50–100; hybrid search in the vector DB |
| generation | GGUF/AWQ quantization + vLLM continuous batching on owned GPUs | GPT-4o / Claude / Gemini APIs — inference in tens to hundreds of ms |
The fork that decides your architecture is the vector index, because it's the one where the hack and the product differ by an order of magnitude:
Hack: Shard It Yourself
- Disk-based ANN — Annoy or Hnswlib load indexes from disk, trading latency for RAM headroom.
- Manual sharding — custom routing or fan-out-and-merge across machines; every shard-split is your on-call problem.
- Metadata in self-hosted Redis or sharded Postgres — consistency between vectors and text is on you.
Paid: Managed Vector DB
- Milvus benchmark: 1 billion vectors at 99% recall in 10 ms — with replication and backups built in.
- Meta-scale precedent: ANN indexes over billions of document vectors serving retrieval at sub-100 ms.
- Hybrid search + filtering come free — the features you'd otherwise hand-build in the retrieval fork.
For generation, the same asymmetry: Cloudflare reports LLM inference in tens of milliseconds for small models and hundreds for large ones on Workers AI — latencies a self-hosted vLLM stack only matches after serious GPU spend and MLOps investment. The hack (quantization + continuous batching) is a real path, but you are signing up to operate an inference company on the side.
Evaluation: Offline Scores, Online Drift
Offline, measure retrieval with Recall@k, MAP, and NDCG against a labeled query set, then score the end-to-end system with RAGAS — four metrics, each of which localizes a different failure:
| RAGAS metric | Question it answers | Failure it catches |
|---|---|---|
| faithfulness | Is the answer factually consistent with the retrieved context? | Hallucinations that sound right but cite nothing. |
| answer relevance | Does the answer actually address the question? | Fluent non-answers and topic drift. |
| context relevance | Was the retrieved context on-topic for the query? | Retriever noise crowding out signal. |
| context recall | Is everything the ground truth needs present in the context? | Chunking that splits the answer in half. |
Online, three monitors run continuously — because a continuously updated corpus means the ground truth itself moves:
System performance
Per-component latency (embed, search, LLM), throughput, and error rates — Prometheus + Grafana for metrics, Jaeger for traces across the pipeline.
Quality proxies
Thumbs up/down on answers, repeated queries, and follow-up rates — cheap online signals that a RAGAS regression is reaching users.
Data & concept drift
Watch the embedding distribution of incoming documents and the relevance scores of retrieved chunks — anomalies mean new topics or shifted user intent.
The Edge Cases That Separate Answers
Long documents
Hierarchical chunking (document → section → paragraph) with recursive retrieval: find the relevant section first, then the chunks inside it. Multi-query generation lifts recall on complex questions.
Out-of-domain queries
Threshold on retrieval confidence and answer “I don't have enough information” below it. A grounded refusal beats a fluent hallucination every time.
Real-time updates
Stream document changes through Kafka into the embedding pipeline; accept eventual consistency on index updates rather than blocking queries on rebuilds.
Cost controls
Cache query embeddings and LLM answers for common queries; batch embedding and inference; route easy questions to smaller fine-tuned models.
Phase 4: The Final Boss — a $427,700 Invoice
Take Path B everywhere at enterprise scale — 100M documents, ~100M queries/month — and price it. The striking part isn't the total; it's the shape:
ingestion 100K docs × 10 pages × $1.50/1K pages (Textract) = $1,500
+ Lambda & S3 = $200
embeddings 10M updated chunks × 500 tok × $0.2/1M tok = $1,000
vector DB 100M vectors, 10K RPS (managed, enterprise) ≈ $15,000
reranker 100M queries × $0.10/1K reranks = $10,000
LLM 50B input tok × $5/1M = $250,000
10B output tok × $15/1M = $150,000 = $400,000
───────────────
total ≈ $427,700/month“94% of the bill is one component. Every optimization that doesn't reduce LLM tokens is rounding error.”
This is also why the reranker pays for itself: $10,000/month of Cohere Rerank that trims the context from 10 sloppy chunks to 3 precise ones cuts input tokens on a $250,000/month line item — and improves faithfulness at the same time. In RAG, cost optimization and quality optimization are frequently the same change.
Takeaway
References & Further Reading
Practice This Question
This post is adapted from our interview question bank — the original comes with structured prep, the full four-phase grounded design, and space for your own notes.