Testing Agent Skills Systematically with Evals | OpenAI Developers

This post introduces a systematic approach to testing and evaluating agent skills by treating them as prompts for LLM applications. It details a pattern for creating evals that include defining success criteria (outcome, process, style, efficiency goals), crafting `SKILL.md` files with clear names and descriptions, manually triggering skills to identify assumptions and edge cases, and using targeted prompt sets with deterministic graders. The post emphasizes the use of `codex exec --json` to capture detailed run traces for automated scoring, enabling engineers to measure skill improvements and regressions objectively. It provides concrete examples of skill definitions, prompt sets for evaluation, and the structure of deterministic checks.