Agentic Research and Discovery Loops

14 videos · Sep 2026 – Oct 2026

What this trail covers

This post contributes to the field of agentic research by detailing experiments with AI agents (ARIA) tasked with optimizing AI model training and research workflows. It highlights the potential for agents to automate the launch, monitoring, summarization, and reporting of experiments, significantly accelerating the research process. The work also emphasizes the importance of agent-focused observability, treating evals as CI, and human review of traces for effective productionization of AI agentic systems. This installment specifically details multi-session fan-out with custom tools, scheduled agent deployments, and a research preview called 'Dreaming' that improves memory stores by reviewing past sessions.

All AI Engineer trails

Oct 2026Build a Research Agent with Claude Managed Agents (Part 2 of 4) — AnthropicThis post details the construction of a research agent using Anthropic's Claude Managed Agents. It extends a console agent to a full research desk by incorporating skills (e.g., SEC EDGAR), subagents, custom tools, and a shared memory store. A key feature is the implementation of outcome-graded analysis, where a grader evaluates the agent's work against a rubric, prompting retries until a passing grade is achieved. The discussion also touches upon vaults for secret management, the distinction between the Agent SDK and managed agents, guardrails for agent loops, and shared memory mechanisms.
Oct 2026Scheduled Agents, Dreaming and Q&A: Claude Managed Agents (Part 3 of 4) — AnthropicThis post details advancements in Claude Managed Agents, focusing on multi-session fan-out with custom tools, scheduled agent deployments, and a research preview named 'Dreaming' that enhances memory stores by analyzing past sessions. It also covers Q&A on observability, long-running agents, workspaces for tenancy, skills vs. multi-agent systems, and integration with Claude Code and AWS.
Oct 2026Hill-Climbing Skills: Improve Agents Without Changing the Model — Shubhankar Srivastava, BrowserbaseThis talk introduces a 'hill-climbing' approach to improve AI agent skills without modifying the underlying model weights. It details an 'AutoBrowse' loop where a 'doer' agent interacts with a web environment (e.g., Google Flights, conference schedules, restaurant searches), and a 'teacher' agent analyzes screenshots, traces, and results to generate 'lessons'. These lessons are stored in a reusable 'strategy file' which is then used to guide subsequent agent attempts, effectively learning from past failures and successes. The talk demonstrates how this process can discover URL shortcuts, optimize navigation, and handle site-specific constraints, leading to more efficient and reliable task completion. It also discusses the requirements for applying this method to other domains (replayable environments, verifiable outcomes) and addresses challenges like prompt injection, skill maintenance, and defining optimization objectives.
Oct 2026Parameter Golf with AutoResearch — Vayum Arora, Zhengyao Jiang, Dixing Xu & Dhruv Srikanth, Weco AIThis talk introduces 'Parameter Golf with AutoResearch' as a method to streamline AI model training experiments. It details Weco AI's approach of using an agent to retain code changes that improve a language model's score and discard others. The presentation covers 'Mini Parameter Golf' for making training experiments inspectable during execution, the Aiden system for proposals and evaluations, and a practical setup combining a modifiable source file, a fixed evaluation command, and a measurable goal. Key aspects discussed include fast and stable evaluations, clear metric direction, time limits, initial guidance, protection against reward hacking, and an inspectable experiment tree for steering research. The Flappy Weco competition is used as an example to demonstrate designing evaluations that reward desired behavior.
Oct 2026From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & BiasesThis post contributes to the field of agentic evaluation by introducing the WolfBench framework, which addresses the limitations of single-average scores for agent performance. It details a comprehensive evaluation pipeline that includes six layers: benchmark, runner, sandbox, agent, model service, and settings. The framework emphasizes repeatable runs under uniform conditions to expose reliability gaps, comparing average, best, and worst runs, tasks solved at least once, and tasks solved every time. It also highlights critical considerations such as token usage and total cost, the trade-off between reasoning effort and score, distinguishing infrastructure failures from task failures, and the nuances of zero-temperature runs. The workflow integrates tools like Harbor, Marimo, and Weave for launching, filtering, and inspecting agent runs, providing an interactive leaderboard that visualizes consistency and cost alongside headline scores.
Advertisement
Oct 2026Small Models, Big Results: Training a Finance Agent for Under $500 — Charles Dickens, Snorkel AIThis post details the training of a 4 billion parameter model (Qwen3 4B) using reinforcement learning (RL) with the rLLM framework for under $500, which outperformed a 235 billion parameter model on financial question-answering tasks (FinQA dataset). The key insight is that disciplined tool use, rather than raw reasoning depth, was the bottleneck. The FinQA dataset was built from SEC 10-K filings with expert validation. The training utilized simple pass/fail rewards and demonstrated that specialized agents can be trained cost-effectively, with skills transferring to harder problems without degrading general tool use. This provides a blueprint for building specialized enterprise AI agents.
Sep 2026The Death of the Code Review: What the Data Actually Says — Laurie Voss, Arize AIThis post analyzes the impact of AI agents on software development workflows, specifically focusing on code generation and review. It presents data indicating that while AI agents dramatically increase code output (741%), the increase in shipped software is significantly lower (30%), identifying human review as the primary bottleneck. The talk critiques traditional code review effectiveness, citing studies showing reviewer performance degradation beyond 400 lines and the emergence of extremely large PRs from agents. It explores current industry practices, including OpenAI's zero-human-code product, mergeability benchmarks (SWE-bench), and the integration of review into tools like Cursor and GitHub Copilot. The post argues that mergeability itself is becoming a training signal for frontier models and that the human checkpoint is shifting towards production as the final arbiter, with prompt injection posing a challenge to automated reviewers. The core technical contribution is the analysis of how AI agents are forcing a re-evaluation and systemic rebuilding of the code review process, moving towards automated review harnesses and production-level validation.
Sep 2026We Let Claude Code and Codex Race Human Researchers — Elie Bakouch, Prime IntellectThis post details an experiment where AI agents (Claude Code, Codex) were used to compete in an 'Optimizer Speedrun' to train a GPT-2 level model, with both agents surpassing the human record. It observes distinct behaviors between agents: Claude Code's tendency to stop and declare records unbreakable versus Codex's continuous operation and extensive note-taking. The experiment also explored a six-day run with Claude, Codex, Kimi, and GLM, measuring efficiency in tokens and identifying that Kimi was the most token-efficient. A key finding is that none of the models invented new optimizers, instead combining existing ideas. The post introduces Prime Intellect's ongoing work on an AlphaEvolve-style loop for AI-driven discovery and argues for open research in this domain.
Sep 2026Beating RL With Reflection: GEPA and Optimize Anything — Lakshya A. Agrawal, GEPAThis post introduces GEPA (Generative Exploration and Prompting Agent) and Optimize Anything, frameworks that leverage reflective optimization for AI tasks. GEPA significantly improves sample efficiency in reinforcement learning by having an AI model read full traces (thoughts, tool calls, errors) and write better prompts, outperforming traditional RL algorithms like GRPO with far fewer examples. Optimize Anything extends this to any text-based, scoreable task, demonstrating dramatic improvements in agent performance across various benchmarks (e.g., ARC-AGI, AMD NPU coding, GPT-5 mini coding). The work also touches on learning evals from production traces and co-optimizing prompts and weights.
Sep 2026Long-Horizon Agents Need Experiments, Not Just Prompts — Erina KaratiThis post introduces an autoresearch loop framework for improving the long-horizon behavior of AI agents. The framework addresses the breakdown of agent memory and decision-making over extended periods by wrapping agents in a controlled experimental loop. Key components include: running controlled scenarios (e.g., spreading facts, rumors), collecting detailed traces, evaluating performance on a balanced scorecard (reach, source retention, uncertainty preservation, replanning, privacy), and iteratively improving a small, frozen policy surface. The post argues that memory alone is insufficient for long-horizon agents and that rollback mechanisms and experimental validation are critical, extending these lessons beyond game agents to broader AI engineering applications.
Sep 2026How We Built an Agent That Improves Itself — Zubin Aysola, Weights & BiasesThis post details the engineering approach to building an AI agent (ARIA) that can improve itself. Key technical contributions include: 1. A robust evaluation framework where production traces are converted into offline evaluation tasks, ensuring airtight measurement as benchmarks, evals, and agent configs evolve together. 2. Maintaining byte-for-byte identical production and research agents to ensure accurate comparisons. 3. Defining agent variants in YAML for side-by-side execution and testing. 4. Providing agents with an unconstrained sandbox for experimentation. 5. A scoring mechanism that uses both pass/fail and relative comparison metrics. 6. Generating a large number of evaluation tasks (886 in this case) by treating every production miss and win as a new task, creating a self-improving flywheel. 7. The use of W&B Weave for production and offline tracing to capture detailed execution information. 8. A model-agnostic harness and YAML variants for flexibility in agent configuration and testing.
Sep 2026An AI Research Agent That Runs Your Experiments — Tim Sweeney, Weights & BiasesThis post details the development and demonstration of ARIA, an AI research agent by Weights & Biases, designed to automate AI experiment execution and analysis. It showcases ARIA's ability to launch and monitor training jobs, summarize results, identify patterns across hundreds of experiments, and generate reports and dashboards. The technical architecture involves logging traces to Weave, using LLM judges on live traffic for signal detection, defining tasks as YAML unit tests, and employing a nightly evaluation suite for go/no-go decisions. Key takeaways for productionizing agents include investing in agent-focused observability, treating evals as CI, maintaining human review of traces, and providing context and tools before over-engineering the harness.
Sep 2026The Loop Is the Product — Roland Gavrilescu, IntrospectionThis post introduces a blueprint for autoresearch in 2026, centered on three core ideas: 1) 'The loop is the product,' emphasizing the quality of signals and verifiers within agentic loops and how their outputs feed subsequent loops. 2) 'System distillation is the moat,' proposing the capture of lessons learned from agent loops (including evals, judges, skills, and human judgment) into portable, versioned 'agent recipes' that are owned by the creator, independent of models or providers. Introspection is releasing an early version of these recipes called Pi recipes. 3) 'Valued work per watt' as the primary optimization score, demonstrated through a talent-sourcing agent example that illustrates spotting patterns in traces, calibrating judges with human-in-the-loop feedback, and A/B testing user taste in production before promoting changes.
Sep 2026AI-Generated Code Is Already Competing With Human Code — Daksh Gupta, GreptileThis talk provides empirical evidence on the quality of AI-generated code in enterprise codebases, analyzing over a million pull requests. It demonstrates that AI-generated code (from models like Codex, Claude Code, and Devin) performs comparably to human-written code across metrics like revert rates, issue severity (P0/P1/P2), and review rounds. Notably, humans were more likely to introduce P0 bugs. The talk also details specific failure modes for different AI models (e.g., SQL injection for Claude, auth bypass for Devin, N+1 queries for Cursor) and discusses the implications for code review at scale, proposing a new validation framework based on user contract adherence, future violation likelihood, and author intent, implemented using blast-radius analysis and sandboxed browser agents.