Introducing Bloom: an open source tool for automated behavioral evaluations

Introduced Bloom, an open-source agentic framework for automated behavioral evaluations of frontier AI models. Bloom generates targeted evaluation suites by analyzing behavior descriptions, ideating scenarios, rolling them out with simulated users/tools, and using a judge model for scoring. It correlates strongly with human judgment and reliably separates baseline from misaligned models. The post details the four-stage evaluation process (Understanding, Ideation, Rollout, Judgment) and demonstrates its utility by replicating known evaluations and discovering new insights into model behavior, such as the reduction of self-preferential bias with increased reasoning effort. Bloom integrates with Weights & Biases and exports Inspect-compatible transcripts.