How we built AEO tracking for coding agents

This post details the engineering effort to build a system for tracking and analyzing the behavior of coding agents. Key contributions include: 1. Designing a six-step lifecycle for running agents in isolated Vercel Sandbox MicroVMs (creation, CLI installation, credential injection, execution, transcript capture, teardown). 2. Implementing a configuration-driven approach to define agents and their setup/execution commands. 3. Routing agent LLM calls through Vercel AI Gateway for unified logging, rate limiting, and cost tracking by overriding provider base URLs via environment variables. 4. Developing a four-stage normalization layer to handle diverse agent transcript formats: transcript capture (agent-specific), parsing (tool name and message shape normalization), enrichment (extracting structured metadata), and aggregation into a unified brand pipeline. Specific technical details include handling JSONL and stdout transcripts, mapping agent-specific tool names to canonical ones, and flattening varied message structures.