Designing AI resistant technical evaluations

What this post added

This post details the evolution of Anthropic's performance engineering take-home test, specifically how it was repeatedly defeated by increasingly capable AI models (Claude Opus 4, then Opus 4.5). It describes the original design, the simulated accelerator environment with features like manual scratchpad memory, VLIW, and SIMD, and the parallel tree traversal problem. It then outlines the iterative redesign process, including removing multicore parallelism, shortening the time limit, and introducing new machine features to increase depth and complexity. The post highlights the challenge of designing evaluations that remain effective with AI assistance and the ongoing effort to find creative ways to distinguish human skill.