Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse

This post details the practical implementation of an AI engineering loop for continuous improvement of AI agents, specifically a support agent named Specs. It demonstrates how to build useful traces by connecting a TypeScript application using the OpenAI Agents SDK to Langfuse. Key contributions include setting up judge evaluators for user disagreement and code-based issues (like all caps), configuring scope checks against system prompts, and using these evaluators to score production traces. The post shows how to seed production traces, use the Langfuse CLI and a coding agent to investigate patterns (retrieval, retries, latency, evaluation coverage), and propose code changes based on these findings. It emphasizes turning observed failures into representative datasets for offline experiments and closing the loop for continuous improvement.