How Clay Monitors Millions of Agent Runs

This post details how Clay monitors millions of agent runs by leveraging LangSmith's insights product and online evaluators. It addresses the challenge of manual review becoming infeasible at scale and explains how these tools provide visibility into agent behavior. The core contribution is the practical application of LangSmith for large-scale agent monitoring and evaluation, enabling engineers to understand and improve agent performance without manual inspection of every run.