Measuring time savings from Figma Make

This post details the methodology and results of a randomized controlled trial (RCT) designed to measure the time savings and ease of use provided by Figma Make. It explains the challenges of measuring AI impact due to confounders and why traditional methods like A/B testing and causal inference on log data were insufficient. The post outlines the RCT design, including participant selection (50 designers, 50 PMs), task design (social media post editing, dark mode conversion, adding menu options, comment fly-up interaction), and moderation script. Key findings include an overall 20% reduction in time to task completion, 16% improvement in task ease, and 15% improvement in perceived usability. Product Managers saw greater gains (23% time savings) and benefited more from simpler tasks, while Product Designers saw improvements on more complex tasks. The post also discusses the use of hypothesis testing and OLS regression for analysis.