Design RAG Evaluation and Debugging
Design RAG Evaluation and Debugging
You are responsible for a production RAG-powered semantic search feature within a fintech product. This feature allows users to ask natural-language questions, retrieves relevant internal documents or records, and then uses an LLM to generate an answer with inline citations.
Your task is to design a lightweight yet production-ready evaluation and debugging framework for this system. The focus is on your ability to measure quality at each stage, prevent regressions before deployment, and pinpoint live failures with concrete evidence.
The problem is structured into five parts, each to be discussed coherently, ensuring that offline metrics align with CI gates and dashboards, and that debugging walkthroughs reference proposed instrumentation.
Constraints & Assumptions
- Domain: Fintech (banking, payments, compliance). Inaccurate or unsupported answers pose significant regulatory and trust risks; faithfulness and permission-correctness are paramount.
- Scale: Approximately 10^5 – 10^6 queries per day. The document corpus consists of hundreds of thousands to a few million documents/records, with daily updates (policies and account data change frequently).
- Pipeline: Query normalization → optional query rewrite → query embedding → vector (ANN) retrieval (optionally hybrid with lexical/BM25) → metadata/permission filtering → reranking → context construction/truncation → LLM generation → citation extraction → logging/feedback.
- Multi-tenancy: Documents are tenant-specific; retrieval must enforce permission and tenant boundaries.
- "Lightweight": Prioritize a concise, maintainable set of metrics and automated checks. Leverage production logs rather than building an extensive labeling organization.
Part 1 — Offline Evaluation
Define metrics for retrieval quality, answer quality, faithfulness, latency, and cost as computed offline on a fixed benchmark. For each metric, describe what it measures and a target threshold. Clearly distinguish between metrics evaluating individual stages and end-to-end performance.
Part 2 — Online Evaluation
Describe online signals: product metrics, business metrics, user feedback, A/B testing methodologies, and guardrail metrics. Explain the limitations of offline metrics alone and how to conduct a controlled experiment for retriever or prompt changes.
Part 3 — Evaluation Datasets
Explain the origin of benchmark data, label creation processes, generation and use of hard negatives, and strategies for keeping the benchmark fresh while maintaining trend comparability.
Part 4 — Automation, Gates, Dashboards & Alerts
Detail how evaluation integrates into CI/CD, define model-release gates, outline essential dashboards, specify alerting criteria, and describe regression analysis procedures when metrics shift.
Part 5 — Failure Analysis: Sudden Drop in Semantic Search Quality
Given a sudden drop in production search relevance, walk through a concrete debugging process. This includes instrumentation, failure localization, ranking likely root causes, validation experiments, the fix, and proof of the fix's efficacy with evidence, attributing the regression to a specific stage.
Similar LeetCode problem: Evaluating RAG Systems
AI interview prep
Core dilemma
Balancing the need for comprehensive, trustworthy evaluation of a complex, multi-stage ML system (RAG) against the constraints of "lightweight", production-readiness, and cost-effectiveness, especially in a high-stakes domain like fintech.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design RAG Evaluation and Debugging
Design RAG Evaluation and Debugging
You are responsible for a production RAG-powered semantic search feature within a fintech product. This feature allows users to ask natural-language questions, retrieves relevant internal documents or records, and then uses an LLM to generate an answer with inline citations.
Your task is to design a lightweight yet production-ready evaluation and debugging framework for this system. The focus is on your ability to measure quality at each stage, prevent regressions before deployment, and pinpoint live failures with concrete evidence.
The problem is structured into five parts, each to be discussed coherently, ensuring that offline metrics align with CI gates and dashboards, and that debugging walkthroughs reference proposed instrumentation.
Constraints & Assumptions
- Domain: Fintech (banking, payments, compliance). Inaccurate or unsupported answers pose significant regulatory and trust risks; faithfulness and permission-correctness are paramount.
- Scale: Approximately 10^5 – 10^6 queries per day. The document corpus consists of hundreds of thousands to a few million documents/records, with daily updates (policies and account data change frequently).
- Pipeline: Query normalization → optional query rewrite → query embedding → vector (ANN) retrieval (optionally hybrid with lexical/BM25) → metadata/permission filtering → reranking → context construction/truncation → LLM generation → citation extraction → logging/feedback.
- Multi-tenancy: Documents are tenant-specific; retrieval must enforce permission and tenant boundaries.
- "Lightweight": Prioritize a concise, maintainable set of metrics and automated checks. Leverage production logs rather than building an extensive labeling organization.
Part 1 — Offline Evaluation
Define metrics for retrieval quality, answer quality, faithfulness, latency, and cost as computed offline on a fixed benchmark. For each metric, describe what it measures and a target threshold. Clearly distinguish between metrics evaluating individual stages and end-to-end performance.
Part 2 — Online Evaluation
Describe online signals: product metrics, business metrics, user feedback, A/B testing methodologies, and guardrail metrics. Explain the limitations of offline metrics alone and how to conduct a controlled experiment for retriever or prompt changes.
Part 3 — Evaluation Datasets
Explain the origin of benchmark data, label creation processes, generation and use of hard negatives, and strategies for keeping the benchmark fresh while maintaining trend comparability.
Part 4 — Automation, Gates, Dashboards & Alerts
Detail how evaluation integrates into CI/CD, define model-release gates, outline essential dashboards, specify alerting criteria, and describe regression analysis procedures when metrics shift.
Part 5 — Failure Analysis: Sudden Drop in Semantic Search Quality
Given a sudden drop in production search relevance, walk through a concrete debugging process. This includes instrumentation, failure localization, ranking likely root causes, validation experiments, the fix, and proof of the fix's efficacy with evidence, attributing the regression to a specific stage.
Similar LeetCode problem: Evaluating RAG Systems