AI Agent Evaluation Framework

Together AI is developing and promoting novel evaluation frameworks for AI agents, focusing on their ability to predict future events. This approach, termed FutureBench, draws from real-world prediction markets and emerging news to create interesting prediction tasks grounded in actual future outcomes. It addresses data contamination issues inherent in traditional benchmarks by making predictions about future events, which are inherently unverifiable at training time. The framework operates on three levels: framework comparison, tool performance, and model capabilities, allowing for systematic evaluation of agentic systems. Initial results show agentic models performing better than simple language models, with different models exhibiting distinct information gathering strategies.