The Self-Improving OSS Agent Stack — Marc Klingen, Langfuse

This post details how to automate the AI agent improvement loop, moving from manual trace analysis to AI-driven proposal of fixes, dataset maintenance, and evaluation criteria updates. It introduces the concept of an 'online + offline agent loop' that connects tracing and monitoring with datasets, experiments, and evals. The talk emphasizes the importance of owning trace data for agents to effectively read and learn from, and demonstrates a coding agent using Langfuse's skills to identify and fix internal jargon in a changelog writer, including back-testing improvements before shipping.