Design a RAG Q&A Agent over Law Firm Legal Memos
Design a RAG Q&A Agent over Law Firm Legal Memos
Design a retrieval-augmented generation (RAG) AI agent that answers legal questions by grounding its responses in published memos and client alerts from large law firms. The system should crawl and ingest these memos, retrieve the most relevant passages for a natural-language question, and produce a grounded, citation-backed answer. Answers not supported by indexed memos should be refused rather than hallucinated.
Corpus: Approximately 500 large law firms, each publishing 5-50 memos/week. Steady state of 1-5 million memos, growing by tens of thousands per week. Memos are primarily HTML and PDFs (500-5,000 words each) in English.
Freshness: Newly published memos should be answerable within a few hours.
Query Load: Starts at ~10 queries/second peak, designed to scale to ~100 queries/second.
Latency: p95 ≤ ~5 seconds for end-to-end answers; token-by-token streaming is allowed.
Correctness: High bar for accuracy; fabricated or uncited answers are unacceptable. Every factual claim must be attributable to a retrieved passage.
Assumptions: Use of hosted LLM and embedding models is permitted. Per-token API costs are a consideration at scale. Only publicly published memos are ingested, respecting robots.txt and terms of service.
The design should cover the entire system: crawling/ingestion pipeline, chunking and embedding/index design, retrieval layer, answer-generation layer (LLM prompting, grounding, citations), and offline/online evaluation and monitoring. Explicitly address the trade-off between retrieval quality (indexing/recall) and system scale.
AI interview prep
Core dilemma
Balancing the need for comprehensive, accurate, and up-to-date legal information retrieval with the inherent complexities and costs of processing a massive, heterogeneous corpus of legal documents at scale, while ensuring strict factual grounding and citation to avoid legal malpractice.
4 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design a RAG Q&A Agent over Law Firm Legal Memos
Design a RAG Q&A Agent over Law Firm Legal Memos
Design a retrieval-augmented generation (RAG) AI agent that answers legal questions by grounding its responses in published memos and client alerts from large law firms. The system should crawl and ingest these memos, retrieve the most relevant passages for a natural-language question, and produce a grounded, citation-backed answer. Answers not supported by indexed memos should be refused rather than hallucinated.
Corpus: Approximately 500 large law firms, each publishing 5-50 memos/week. Steady state of 1-5 million memos, growing by tens of thousands per week. Memos are primarily HTML and PDFs (500-5,000 words each) in English.
Freshness: Newly published memos should be answerable within a few hours.
Query Load: Starts at ~10 queries/second peak, designed to scale to ~100 queries/second.
Latency: p95 ≤ ~5 seconds for end-to-end answers; token-by-token streaming is allowed.
Correctness: High bar for accuracy; fabricated or uncited answers are unacceptable. Every factual claim must be attributable to a retrieved passage.
Assumptions: Use of hosted LLM and embedding models is permitted. Per-token API costs are a consideration at scale. Only publicly published memos are ingested, respecting robots.txt and terms of service.
The design should cover the entire system: crawling/ingestion pipeline, chunking and embedding/index design, retrieval layer, answer-generation layer (LLM prompting, grounding, citations), and offline/online evaluation and monitoring. Explicitly address the trade-off between retrieval quality (indexing/recall) and system scale.