The Autonomous Computer: Infrastructure for Computer Use Agents — Ang Li, Simular

This talk introduces the concept of 'autonomous computers' and the infrastructure required for computer use agents. It demonstrates an agent (Agent S) that can perform tasks by combining adaptable vision agents with reusable automation code, checking the environment at each step. The research trajectory includes visual grounding, multiple models, and parallel attempts, with a focus on reliability across repeated runs with varying prompts and environments. The infrastructure discussion extends from personal desktops to fleets of cloud computers, with examples of legacy form entry and Android interactions. Key challenges highlighted include making everyday computer use reliable, understanding screen content, verification loops, irreversible actions, OS restrictions, and adapting to application changes.