Designing AI resistant technical evaluations

This post details the evolution of Anthropic's take-home test for performance engineers, specifically how it was redesigned three times to remain effective against increasingly capable AI models (Claude Opus 3, 4, 4.5). It describes the original take-home design simulating an accelerator with features like manual scratchpad memory, VLIW, SIMD, and multicore, used for a parallel tree traversal. It highlights how AI models defeated earlier versions by solving the problems within the allotted time, necessitating the introduction of more complex optimization insights, reduced time limits, and the exploration of AI-resistant problem structures that require deep system understanding and creative workarounds beyond current AI capabilities. The post also discusses the challenges of designing AI-resistant evaluations that still allow for AI assistance on the job.