Direct agents with visual prompts in Design Mode
Jun 5, 2026Erik Nilsson, Ian Huang & Ryo Lu

Summary
This post updates the Design Mode capability by introducing new ways for users to direct agents with visual prompts. It enhances the existing visual editor by allowing users to point at elements, draw on the interface, or use voice narration to convey design changes. The system now captures element identity (xpath, component, attributes, computed styles, props) and spatial context (screenshot) to provide agents with richer information for code editing. It also supports multitasking by enabling users to send multiple edits concurrently, improving the iteration loop for UI refinement.