Llm Observability And Monitoring

8 videos · Oct 2026

What this trail covers

This post contributes to the field of LLM observability and evaluation by detailing the challenges of traditional testing methods for AI systems and proposing a robust evaluation framework. It emphasizes moving beyond simple assertions and fuzzy matching to sophisticated LLM judges, curated datasets of real-world scenarios, and continuous calibration against human verdicts. The work highlights the importance of integrating live policy retrieval (e.g., via OpenRAG) and dynamic model refinement in production.

All AI Engineer trails

Oct 2026Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouseThis post details the practical implementation of an AI engineering loop for continuous improvement of AI agents, specifically a support agent named Specs. It demonstrates how to build useful traces by connecting a TypeScript application using the OpenAI Agents SDK to Langfuse. Key contributions include setting up judge evaluators for user disagreement and code-based issues (like all caps), configuring scope checks against system prompts, and using these evaluators to score production traces. The post shows how to seed production traces, use the Langfuse CLI and a coding agent to investigate patterns (retrieval, retries, latency, evaluation coverage), and propose code changes based on these findings. It emphasizes turning observed failures into representative datasets for offline experiments and closing the loop for continuous improvement.
Oct 2026Why 80% Reliability Isn't Good Enough — Felipe Blanes, Amazon AGI LabThis post introduces the concept of the 'benchmark illusion' in AI agent development, where static evaluations fail to capture real-world performance degradation due to unexpected customer usage. It proposes a customer-centric 'eval flywheel' as a solution, comprising four steps: defining success from the customer's perspective, capturing real-world signals (including direct customer feedback), diagnosing gaps in the model, harness, or product, and using these insights to inform development decisions. The talk also highlights the 'trust cliff,' where reliability below a certain threshold (around 92%) erodes customer trust, and advocates for transparency about model limitations over inflated benchmark scores. It shares surprising use cases from Amazon Leo, Hertz, and Sola, and outlines four principles for maintaining fresh and relevant evaluations.
Oct 2026Why Building an Eval Platform Is Harder Than It Looks — BraintrustThis post details the evolution of AI agent evaluation platforms, moving from simple spreadsheets to sophisticated production flywheels. It highlights the challenges of non-deterministic LLMs necessitating both pre-production evals and post-production observability. A key contribution is the introduction of BTQL, a specialized query language designed to handle the massive and complex data generated by agent traces, which are unsuitable for traditional data warehouses. The talk also explores the concept of agents running their own evals and identifying novel failure modes.
Oct 2026Advanced Workshop: Mastering AI Observability — Doug Guthrie, BraintrustThis workshop details a practical workflow for improving AI agents using production feedback. It starts with a Python support agent built on the OpenAI Agents SDK, adding tracing to establish an offline evaluation baseline and publishing custom scorers. Online automations grade incoming traces, and Topics groups patterns in tasks, sentiment, and workflow issues. A custom facet targets failures missed by default classification. The process involves configuring conversation grouping and sampling, importing sample traces, and inspecting labels and topic maps for investigation. A Loop assistant queries trace data with SQL for analysis. Selected failures are returned to evaluation datasets for checking subsequent changes. A coding agent with the Braintrust CLI and an improvement skill automates repository changes: investigating logs, proposing code and scorer modifications, adding regression examples, and running evaluations. Findings, explanations, and evaluation evidence are packaged into pull requests. The workshop also touches on exposing remote evaluations to a playground for collaborator adjustments.
Oct 2026The Transcript Looked Fine. The Call Wasn't. — Debugging Voice Agents, ArizeThis post contributes to LLM observability by highlighting the specific challenges of debugging voice agents, where text transcripts alone are insufficient. It introduces the necessity of integrating audio, transcript, and trace data into a unified view for debugging. The talk advocates for OpenInference semantic conventions for real-time audio processing across different providers and proposes audio-native evaluations that assess metrics like sentiment, latency, interruptions, and task success directly from the audio. It also outlines an observe-evaluate-improve loop and agent experiments for self-healing voice agents, including monitors, investigation agents, and replayed fixes.
Advertisement
Oct 2026Evals in AI: A Deep Dive — Tejas Kumar, IBMThis post contributes to the field of LLM evaluation by demonstrating the limitations of basic assertion-based tests and fuzzy matching for AI systems. It introduces a practical workflow for building and calibrating LLM judges against datasets of customer scenarios, agent responses, and human verdicts. Key contributions include showcasing how to improve judge accuracy by incorporating live policy data (e.g., refund policies via OpenRAG), refining models based on evaluation disagreements, and establishing CI gates with an 80% agreement threshold. The talk also differentiates between pre-deployment checks and runtime harness protections, advocating for deterministic checks before relying on model judgments.
Oct 2026Your LLM App Returned 200 OK. It Was Still Wrong. — Marina Petzel, DatadogThis post introduces a framework for monitoring generative AI applications beyond traditional golden signals. It details specific metrics for cost (token creep, model drift, uncached calls, tagging), safety (prompt injection, PII leakage, toxicity, jailbreaks), and quality (hallucination rate, relevance, user satisfaction, completeness, RAG retrieval quality). The talk highlights how LLM applications break traditional monitoring assumptions and proposes actionable metrics and strategies for effective observability.
Oct 2026The 5 Levels of Self-Driving Production — Eric Schwartz, TraversalThis talk contributes to the field of LLM observability and incident management by introducing the concept of 'self-driving production' and its five levels of autonomy. It argues that the proliferation of AI-written code exacerbates debugging challenges, rendering traditional observability insufficient for multi-hop root cause analysis. The talk proposes a causal approach to RCA, leveraging production world models and causal search, and presents case studies demonstrating its application in alert intelligence and incident resolution at enterprise scale. It also provides five critical questions for evaluating AI SRE tools.