Parameter Golf with AutoResearch — Vayum Arora, Zhengyao Jiang, Dixing Xu & Dhruv Srikanth, Weco AI

This talk introduces 'Parameter Golf with AutoResearch' as a method to streamline AI model training experiments. It details Weco AI's approach of using an agent to retain code changes that improve a language model's score and discard others. The presentation covers 'Mini Parameter Golf' for making training experiments inspectable during execution, the Aiden system for proposals and evaluations, and a practical setup combining a modifiable source file, a fixed evaluation command, and a measurable goal. Key aspects discussed include fast and stable evaluations, clear metric direction, time limits, initial guidance, protection against reward hacking, and an inspectable experiment tree for steering research. The Flappy Weco competition is used as an example to demonstrate designing evaluations that reward desired behavior.