Grab Bench: Evaluating AI on Grab-shaped production work

This post introduces Grab Bench, a new evaluation harness for AI systems designed to assess AI models on Grab-shaped work. It focuses on evaluating specific contracts such as metric faithfulness, tool-parameter discipline, evidence grounding, safety boundaries, and repository-level behavior. Grab Bench utilizes task plugins for various AI tasks including SQL generation, tool use, profile reasoning, and agentic coding, employing deterministic scorers or LLM judges. The system emphasizes safe case design using synthetic or redacted data, scoring contracts over confidence, and making shortcuts visible through baselines and failure taxonomies. It supports internal reproducibility with teaching and certification artifacts and aims to evolve towards pipeline-like evaluations and easier reuse by other teams.