System Design: Retrieval-Augmented Generation (RAG) Question-Answering System
Design a Retrieval-Augmented Generation (RAG) system capable of answering user questions by leveraging an internal document corpus. This corpus may include various types of documents such as product documentation, company policies, and operational runbooks. A core requirement is that all generated answers must be strictly grounded in the retrieved documents, with clear citations (links, IDs, or references) provided for verification.
Functional Requirements
- Users must be able to submit natural-language queries to the system.
- The system will respond with an answer generated by a Large Language Model (LLM).
- All answers must be demonstrably grounded in the retrieved documents, including explicit citations.
- The system must support a comprehensive document lifecycle: ingestion of new documents, updates to existing documents, and deletion of outdated documents.
- While optional, the design should describe how multi-turn conversational capabilities could be supported.
Non-Functional Requirements
- Latency: The 95th percentile (p95) latency for typical queries must be less than 3 seconds.
- Availability: The system must achieve an availability target of 99.9%.
- Data Privacy: The system must enforce access control, as some documents may be restricted to specific users or teams.
- Quality: The system should minimize LLM hallucinations. A mechanism for evaluating and continuously monitoring the quality of generated answers must be included.
Design Scope
Candidates should cover the following aspects in their design:
- High-level architecture and identification of main components.
- The data ingestion and indexing pipeline, including strategies for document chunking, embedding generation, and metadata management.
- The retrieval strategy, encompassing techniques like top-k selection, filtering, and reranking.
- The prompting and generation strategy, focusing on context window management and ensuring accurate citation generation.
- Choices for storage solutions, specifically for vector databases and metadata stores, along with a scaling approach.
- Considerations for caching, monitoring, evaluation frameworks, and handling various failure modes.
AI interview prep
Core dilemma
Balancing the quality and relevance of retrieved information with the efficiency and cost of LLM generation, while strictly enforcing data privacy and maintaining low latency under dynamic data and query loads.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
System Design: Retrieval-Augmented Generation (RAG) Question-Answering System
Design a Retrieval-Augmented Generation (RAG) system capable of answering user questions by leveraging an internal document corpus. This corpus may include various types of documents such as product documentation, company policies, and operational runbooks. A core requirement is that all generated answers must be strictly grounded in the retrieved documents, with clear citations (links, IDs, or references) provided for verification.
Functional Requirements
- Users must be able to submit natural-language queries to the system.
- The system will respond with an answer generated by a Large Language Model (LLM).
- All answers must be demonstrably grounded in the retrieved documents, including explicit citations.
- The system must support a comprehensive document lifecycle: ingestion of new documents, updates to existing documents, and deletion of outdated documents.
- While optional, the design should describe how multi-turn conversational capabilities could be supported.
Non-Functional Requirements
- Latency: The 95th percentile (p95) latency for typical queries must be less than 3 seconds.
- Availability: The system must achieve an availability target of 99.9%.
- Data Privacy: The system must enforce access control, as some documents may be restricted to specific users or teams.
- Quality: The system should minimize LLM hallucinations. A mechanism for evaluating and continuously monitoring the quality of generated answers must be included.
Design Scope
Candidates should cover the following aspects in their design:
- High-level architecture and identification of main components.
- The data ingestion and indexing pipeline, including strategies for document chunking, embedding generation, and metadata management.
- The retrieval strategy, encompassing techniques like top-k selection, filtering, and reranking.
- The prompting and generation strategy, focusing on context window management and ensuring accurate citation generation.
- Choices for storage solutions, specifically for vector databases and metadata stores, along with a scaling approach.
- Considerations for caching, monitoring, evaluation frameworks, and handling various failure modes.