Back to The Future: Evaluating AI Agents on Predicting Future Events
Jul 17, 2025
Summary
This post introduces FutureBench, a new benchmark for evaluating AI agents' ability to predict future events. It details the methodology for generating prediction questions from news and prediction markets, the technical stack used (DeepSeek-V3, Firecrawl, Tavily), and a three-level evaluation framework (framework comparison, tool performance, model capabilities). The post also presents initial results comparing different LLMs (GPT-4.1, Claude3.7, DeepSeek-V3) and analyzes their action patterns and prediction strategies.
More from the Autonomous Data Scientist Agent trail
- EinsteinArena: Harnessing the collective intelligence of agents in the wild to advance science
- DSGym: A holistic framework for evaluating and training data science agents
- How Together AI Uses AI Agents to Automate Complex Engineering Tasks: Lessons from Developing Efficient LLM Inference Systems
- DeepSWE: Training a Fully Open-sourced, State-of-the-Art Coding Agent by Scaling RL
- From Zero to One: Building An Autonomous and Open Data Scientist Agent from Scratch