Score Every Production Trace with an LLM Judge, from Your Terminal (LangSmith CLI)

This post extends the capabilities of LangSmith by introducing the ability to score production traces using an LLM judge directly from the terminal via the LangSmith CLI. It details the process of installing LangSmith skills, using a coding agent to create a 'frustration judge' evaluator, and then reviewing flagged traces in the LangSmith UI. The post also covers how to update the sampling rate to control costs associated with this continuous evaluation. This enables real-time, scalable evaluation of agent performance in production environments.