TechPrufer
Get started
Airbnb
Airbnb logoAirbnb · Feature trail

LLM Evaluation Infrastructure

2 posts · Jul 2026

What this trail covers

This post details the evolution of Airbnb's LLM evaluation infrastructure to enable rapid iteration. It introduces a four-layer approach: diagnostic framing of evaluation noise (dual indeterminacy), a deterministic evaluation foundation via per-sample caching of references and judge scores, bounded and scoped model mutation using micro-adapters (low-rank LoRA patches) with lifecycle rules (fuse co-triggering, retrain on accumulation, unload unused), and end-to-end validation at the seams. This system allows for same-day turnaround on LLM improvements, moving from weeks to a day.

All Airbnb trails
  1. Eval-driven development: Lessons from evaluating GenAI at scaleThis post extends the LLM Evaluation Infrastructure capability by detailing the practical application of Eval-driven development (EDD) for building trustworthy Generative AI products at scale. It elaborates on the five core principles of EDD, introduces a three-layer evaluation methodology (programmatic checks, LLM-as-a-Judge, and human evaluation) with specific examples and calibration techniques for virtual judges, and outlines a detailed approach for evaluating agentic systems by analyzing step, trajectory, and session levels using observability data. A walkthrough of a support policy AI assistant evaluation is provided.
  2. From weeks to a day: how we made LLM evaluation fast enough to iterate onThis post introduces the concept of 'dual indeterminacy' in LLM evaluation, distinguishing between aleatoric (task ambiguity) and epistemic (model/judge limits) uncertainty. It details the implementation of a per-sample cache for references and judge scores to achieve deterministic evaluation. The core innovation is the 'micro adapter,' a low-rank LoRA patch trained in under an hour for specific bug fixes, layered on existing adapters. Lifecycle rules for managing these micro adapters (fuse co-triggering, retrain on accumulation, unload unused) are also described. Finally, it emphasizes the necessity of end-to-end validation at the seams of the LLM system, even with component-level confidence, to catch emergent behaviors.