DSGym: A holistic framework for evaluating and training data science agents

What this post added

Introduces DSGym, a unified framework for evaluating and training data science agents. DSGym integrates diverse data science evaluation suites behind a single API with standardized abstractions for datasets, agents, and metrics. It expands the scope with novel scientific analysis tasks (90 bioinformatics tasks) and challenging end-to-end modeling competitions (92 Kaggle competitions). DSGym provides trajectory generation and synthetic query pipelines for agent training, demonstrating training a 4B model on 2k generated examples. The framework abstracts code execution behind containers, offers a unified JSON interface, and categorizes tasks into Data Analysis and Data Prediction. New datasets DSBio and DSPredict are introduced. Addresses the 'memorization gap' by filtering tasks solvable without data access and analyzes benchmark performance and failure modes, highlighting domain-grounding errors in scientific tasks. Evaluates agent performance on building ML pipelines via DSPredict.