TechPrufer
Get started
Hex
Hex logoHex · Feature trail

Agent Evals

40 posts · Feb 2023 – Oct 2026

What this trail covers

Hex has evolved its agent capabilities by introducing Evals, a testing workflow inspired by internal practices, now available to users. Evals allow for grading agent work beyond just answers, using an LLM-as-judge to assess tool usage, reasoning, and behavior. Test cases are defined in version-controlled files and run from the Hex CLI. Context changes can be tested safely via Context Previews before publishing. Evals enable measurement and improvement of agent quality and cost, helping to test c. This post details how Hex used evals to drive the development of its 'Quick Edits' feature, achieving a 7x reduction in the AI error rate through a hill-climbing approach. The focus was on minimizing wrong edits over incorrect handoffs, and improvements were largely driven by enhancements to the validation harness around the model rather than prompt engineering. Holdout cases were used to prevent overfitting, and the speed of the eval runs enabled rapid iteration on both the AI and the broader feature.

All Hex trails
  1. Hill climbing to glory: using evals to improve AI error rate by 7xThis post details Hex's application of evals and a hill-climbing methodology to improve the error rate of its 'Quick Edits' AI feature by 7x. Key contributions include: prioritizing the reduction of wrong edits over incorrect handoffs, demonstrating that improvements to the validation harness (handling formatting, rounding, and property validation) were more impactful than prompt engineering, and using holdout cases (40% of the eval suite) to prevent overfitting. The post also highlights the importance of tuning the number of attempts per eval case (10 was found to be optimal) to reduce noise, and the benefit of fast eval runs (minutes, <$10) for rapid iteration on AI and feature development simultaneously.
  2. Context in Analytics: Turning Raw Data Into Actionable InsightThis post elaborates on the critical role of context in making AI analytics trustworthy and scalable. It introduces the concept of 'contextual analytics' as the practice of ensuring metrics carry definitions, lineage, quality signals, and business rules. It argues that AI's inability to inherently understand data meaning makes context the primary bottleneck for AI adoption in analytics. The post details how Hex addresses this through its Semantic Model Agent and Context Studio, enabling the creation and management of a centralized context layer. This layer includes semantic context (metric definitions), lineage context, operational context (quality signals), and policy context. The compounding effect of this shared context is emphasized, where each interaction enhances future answers. The post also contrasts this with traditional tool sprawl and data governance, positioning contextual analytics as an infrastructure for business meaning.
  3. Introducing DataBenchThis post introduces DataBench, a new benchmark designed to evaluate agentic analytics capabilities on realistic data work. It details the motivation for creating DataBench, contrasting it with existing benchmarks like Spider 2.0 and DABstep. The post presents the methodology, including the use of a synthetic business (Shorelane Commerce) and Hex's native Evals functionality. It shares performance results for various models (Opus 5, Claude Fable 5, GPT 5.6 Sol, GPT-5.6 Luna, Sonnet 5) across different task types (Q&A, open-ended delegated decisions, trap failure modes). Key findings include the high performance floor, the steep Pareto frontier for GPT-5.6 Luna, and the models' ability to handle messy data when intent is clear. It also identifies judgment as a key area where models struggle, particularly with trap failure modes requiring intuition and deeper reasoning.
  4. Introducing EvalsIntroduces Evals for Hex Agents, a testing framework that uses an LLM-as-judge to evaluate agent performance on data analysis tasks. Evals allow for grading the agent's reasoning and behavior, not just the final answer, by assessing tool usage, clarification requests, and appropriate refusal to answer. Test cases are defined in version-controlled files and executed via the Hex CLI. The feature supports Context Previews for testing changes before publishing and running configuration sweeps across different LLMs. It aims to make agent quality and cost measurable and improvable, turning agent management into a quantifiable process.
  5. Trust used to be impliedThis post details a strategic evolution of the "Agent Evals" capability, shifting the focus from purely internal debugging to external trust-building and adoption. It highlights how consistent, high scores on evals, particularly when shared with business users, can re-establish the trust that was previously implied with traditional dashboards. The post emphasizes the importance of showcasing improvements and demonstrating reliability for common questions to drive user confidence and adoption, framing evals as a marketing and retention tool for AI agents.
  7. Credits and usage visibility for Hex AgentsIntroduced a credit model for Hex Agents, including monthly credit grants per user seat (Professional, Team, Enterprise, Explorer) and pooled add-on credits with auto top-ups. Implemented an effort-based consumption system that dynamically selects LLM models based on task complexity, context, and resource requirements, aiming for cost efficiency. Enhanced visibility by allowing users to view credit balances and per-prompt costs, and providing admins with historical usage logs, spend limit settings, and credit usage data within the Context Studio. This enables better budgeting, ROI calculation, and informed decisions on context curation and model selection.
  8. Kimi K2.7 in Hex: Opus-level Analytics at a Fraction of the CostThis post details the integration of Kimi K2.7, an open-weight model, into Hex. It presents an evaluation of Kimi K2.7 against Opus 4.7 across various analytical tasks, focusing on cost, speed, and intelligence. The analysis highlights Kimi's strengths in semantically modeled and analytically hard questions, its trade-offs in visualization and contextually hard tasks, and its 'worrier' behavior leading to higher correctness at the expense of speed. The post also discusses the economic implications of token pricing versus task cost in agentic workflows and the challenges of open-weight models in data analytics, particularly regarding rambling and context window limitations. It positions Kimi K2.7 as a cost-effective alternative for many day-to-day analytics tasks.
  9. Introducing Agent TasksThis post introduces Agent Tasks, a new capability that allows scheduling of Hex Agent prompts for automated execution and delivery of results to Slack or email. It enables recurring daily briefings, proactive metric drop detection with root cause analysis, and comprehensive key account updates by combining structured and unstructured data. The feature allows the agent to reference previous runs to highlight changes and shifts over time, providing a more dynamic and consistent experience than static dashboards.
  10. We had to build new evals for FableThis post details the introduction of Claude Fable 5 and its performance on new, more challenging "Frontier" evaluation sets designed to test long-horizon analytical reasoning and complex decision-making, revealing significant improvements over previous models. The post also discusses the "golden workflow" of analytical reasoning and the importance of framing results against semantic ground truth.
  11. How we built a lab to evaluate data agentsThis post details the development of 'The Shoebox,' Hex's internal infrastructure for evaluating data agents. It describes the creation of a synthetic business, 'Shorelane Commerce,' with realistic data debt and complexities to serve as a benchmark for agent performance. The Shoebox facilitates pairwise experiment comparisons between candidate and baseline agent runs, supports custom rubrics and ground truths, and integrates with the local Hex dev stack and shared workspaces for consistent evaluation.
  12. AI Analytics Adoption Risks: 7 Pitfalls to AvoidThis post details 7 pitfalls in AI analytics adoption, focusing on governance, verifiable context, fragmentation, self-serve expectations, semantic layer maturity, feedback loops, and data quality. It highlights how ungoverned AI tools lead to metric drift and security risks, the importance of verifiable context for AI-generated answers, and how AI point solutions can exacerbate fragmentation. It also discusses the maturity path for semantic layers, the need for feedback loops between AI output and context, and prioritizing data quality over model chasing. Hex's unified workspace and features like Threads and Notebook Agent are presented as solutions to these challenges, emphasizing governed context and integrated workflows.
  13. Building Topic Discovery into the Hex Context AgentThis post details the evolution and rebuilding of the topic discovery system within the Hex Context Agent. Initially, a complex ML approach using embeddings and HDBSCAN was considered but deemed too infrastructure-heavy. The system was then rebuilt using a simpler, two-stage LLM approach: first, an LLM proposes topics based on thread summaries, and second, another LLM classifies threads into these proposed topics. This approach was refined by splitting discovery and classification into separate LLM calls, significantly improving topic quality and enabling stable, recurring topic generation. The current system runs topic discovery every 2-3 days for organizations with sufficient thread activity, classifying new threads and incorporating them into an incremental discovery pool for potential reclassification. Future work includes hierarchical clustering, upfront topic classification for proactive context injection, and topic-specific eval suites.
  14. Introducing Context SuggestionsIntroduced Context Suggestions, a feature that automatically generates actionable context improvements from user conversations and analyses. This involves a Review Agent that parses data agent interactions to extract summaries, topic tags, and warnings (context gaps, data limitations, user doubt). These insights are synthesized into 'Suggestions' for improving guides, semantic models, or metadata, creating a self-learning flywheel.
  15. Introducing Context Studio — Curate trusted answers for your whole team, wherever they’re working.This post introduces Context Studio, a new system for managing AI agent context and trust. Key technical contributions include: an observability dashboard for tracking agent adoption and identifying confusion; a Thread inspector for diagnosing agent errors and providing specific improvement suggestions; a multi-layered context curation system (endorsements, semantic models, workspace guides); and a context workbench for staging, testing, and deploying context changes with version control and attribution logs.
  16. Shadow AI governance: strategies to rein in rogue modelsThis post discusses strategies for governing 'shadow AI' (unauthorized AI tool usage) by focusing on building governance into platform architecture, implementing risk-based approval processes, aligning with regulatory frameworks, providing approved alternatives, and creating visibility. It highlights the risks of data exposure and metric inconsistency from ungoverned AI. The post advocates for embedding governance in platforms like Hex, which offers AI-assisted features within a controlled environment, and emphasizes the importance of making governed analytics faster and easier than alternatives.
  17. The Notebook Agent just got even betterThis post details significant enhancements to the Notebook Agent, focusing on its ability to perform advanced analytics tasks. New capabilities include chart styling, input parameters, and single value cells for building interactive analyses. The agent can now organize and tidy up analysis by moving/deleting cells and auto-organizing them into sections, leveraging its understanding of the project's DAG. Debugging is improved with graph-aware context, enabling the agent to traverse the project graph to find and fix issues. These updates aim to make the agent a more powerful tool for complex data work beyond simple code generation.
  18. How the best data teams use the Notebook AgentThis post details how data teams at Ramp, Figma, and Notion are using the Notebook Agent within Hex for analytics. It highlights specific use cases such as planning and building analyses, visualizing metrics, generating summaries for stakeholders, empowering non-data teams, and accelerating report development. The post emphasizes the agent's role as a collaborator and accelerant, reducing time spent on debugging syntax and tedious tasks, and enabling faster iteration and insight generation.
  19. What is AI for data analysis?This post introduces the concept of AI for data analysis, explaining how natural language processing and ensemble machine learning are used to automate data analysis workflows. It details the components of AI data analysis systems, including NLP, ensemble ML, governance, semantic layers, and model explainability. The post also discusses different categories of AI analytics platforms and highlights Hex's approach, emphasizing its integrated workspace for data scientists, analysts, and business users, and its extensibility through Slack and MCP connections. It also touches on real-world applications like conversational self-serve analytics and AI-powered exploration of existing artifacts.
  20. How context engineering makes or breaks AI-driven self-serviceThis post introduces and elaborates on the concept of 'context engineering' for AI agents in analytics. It defines context as business knowledge that enables AI agents to understand specific data environments, distinguishing it from model capabilities. The post details four key principles for effective context engineering: leveraging existing work, curating context flexibly, strategic rollout of self-serve, and employing a hybrid architecture. It also describes four specific tools within Hex for managing context: warehouse metadata, endorsements, rules files, and semantic models. The core contribution is framing the data team's role as shifting from answering one-off questions to building a 'context lake' that fosters a virtuous cycle of learning and self-service.
  21. Hex Agent, everywhere: Introducing Slack integration and MCP supportThis post introduces two new entry points for Hex's agentic capabilities: a native Slack integration and Model Context Protocol (MCP) support. The Slack integration allows users to ask data questions directly in Slack channels, with the agent responding in-thread and providing explanations, charts, and links to Hex projects. The MCP server enables integration with AI assistants like Claude Desktop, Claude Code, and Cursor, allowing them to search Hex projects and initiate analyses. These features aim to make Hex's agentic capabilities more accessible within existing user workflows.
  22. Improving the Notebook Agent: Error ClusteringThis post details the implementation of error clustering using K-Means on agent thinking messages to identify and categorize failure modes within the Notebook Agent. It describes the process of embedding messages, clustering by tool, labeling clusters with LLMs, and visualizing results with t-SNE. Specific findings include issues with Dataframe SQL usage, SQL syntax errors, and cell timeouts. The post outlines the resulting improvements: prompt enhancements for Dataframe SQL, a tool for searching Hex docs, improved handling of cell timeouts with user consultation, and a 'notebook DAG' tool to enhance relational understanding of notebook structure. It also highlights the impact of clear error messages on agent recovery and strategy.
  23. Self-serve analytics is finally real (and that's terrifying)This post details how well-modeled data, documentation, rules files, semantic layers, and reusable analyses serve as crucial context for AI agents, enabling self-serve analytics and empowering the entire company. It emphasizes that AI agents are the best users of data team's work, reading documentation and using endorsed analyses to inform their queries. The post highlights the importance of these contextual elements for building trustworthy systems and enabling self-serve analytics.
  24. Why Analytics Agents Break DifferentlyThis post details the challenges and solutions for context management in analytics AI agents, specifically how they break differently than coding agents. It introduces the concept of treating notebooks as computational graphs rather than file trees, the importance of explicit token boundaries and truncation, and the strategy of informing agents about limitations. The contribution focuses on the engineering insights gained from building Hex's Notebook Agent, particularly around context engineering principles for data flows.
  25. How to Work with Hex's Notebook Agent: A Prompting GuideThis post provides a guide on how to effectively prompt Hex's Notebook Agent. It details the agent's key capabilities (agentic search, building a plan, executing analysis, summarizing results), offers mental models for interaction (structured thinking, conversational prompting, meta prompting, scoping context, specifying analysis methods, providing context for business impact, and business rules), and includes copy-paste prompt templates. The post aims to improve user experience and analytical output when working with the Notebook Agent.
  26. How Hex's Notebook Agent changed pricing analysis with AI-powered data analysisThis post details the author's direct experience using the Hex Notebook Agent for pricing analysis. It highlights the agent's ability to write SQL and Python, join tables, use window functions, and build visualizations based on natural language prompts. The author emphasizes the speed and accessibility of the agent, enabling them to perform complex analysis solo, and the confidence gained by being able to see and edit the agent's work. The post also notes the agent's ability to produce output that was validated by the data team without notes, indicating a significant shift in the product's capabilities.
  27. Introducing the Notebook AgentIntroduced the Notebook Agent, an AI agent integrated directly into the Hex notebook environment. The agent leverages Claude Sonnet 4 and Hex tools to assist users with planning, data searching, creating SQL and Python cells, visualizing data, and summarizing findings. Key features include @ mentioning cells and tables for context, autosaving versions of agent-generated threads, and a cell-by-cell diff view for reviewing and accepting/rejecting generated code. This enhances the iterative and collaborative data analysis workflow within Hex.
  28. I'm sorry, but those are vanity evalsThis post introduces a 'funnel' approach to LLM evaluation, decomposing binary pass/fail metrics into a series of cascading steps. It uses GPT-4.1 as a case study to demonstrate how this method provides actionable insights beyond simple pass rates. Key contributions include: 1. Identifying 'eval pass rate' as a vanity metric and advocating for 'clarity metrics'. 2. Explaining the 'funnel' methodology: decomposing tasks into masked, sequential steps (e.g., RAG retrieval, table selection, column matching, query execution, correct data return). 3. Analyzing GPT-4.1's performance through the funnel, revealing a behavioral change (lowercasing/quoting identifiers) that impacted query execution, and highlighting significant upstream improvements in table selection. 4. Discussing the challenges of real-world semantic accuracy in SQL generation evals compared to syntactic ones.
  29. Data is better with a little magicThis post introduces several significant upgrades to Hex's Magic AI assistant: 1. **Generate mode:** Allows Magic to generate multiple cells (SQL, Python, Chart) at once, enabling complex question answering and new analysis kickstarts. New cells are created as drafts for inspection and validation. 2. **Streamlined prompting:** A unified prompt bar (Cmd+Shift+M) supports multi-line instructions and guides auto-fixes, allowing for both one-click fixes and specific directional guidance. 3. **Mentioning data:** The prompt bar now supports '@ mentioning' datasets (warehouse tables or dataframes) to specify resources for generation, improving prompting efficiency and accuracy. 4. **Improved SQL generation:** Enhancements to metadata retrieval pipelines improve Hex's ability to generate accurate SQL queries, with emphasis on leveraging dbt Docs, warehouse metadata, and Hex's Data Manager for richer context.
  30. Managing MagicThis post introduces the Data Browser (formerly Data Manager) as a metadata tool to enhance Hex Magic's SQL generation capabilities. It details how the Data Browser improves table retrieval accuracy and reduces string hallucinations by allowing users to provide custom metadata, schema information, dbt documentation, and endorsed statuses. The post explains the underlying mechanism of vector embeddings and provides practical examples of how to use the Data Browser to exclude irrelevant tables, prioritize correct tables, and add nuanced natural language descriptions to guide the AI's SQL generation. It also highlights how custom descriptions are made visible to human users for improved understanding.
  31. Building a Text-to-SQL ChatbotThis post details the construction of a text-to-SQL chatbot by integrating Hex, LangChain, and OpenAI. It showcases how to leverage natural language to query databases, with specific emphasis on using user selections to dynamically provide context (database, schema, column names, descriptions, data types) to the LangChain and OpenAI functions, thereby improving bot accuracy. The post also mentions the use of LangSmith for prompt tracing and future considerations like incorporating memory and chat features.
  32. Magic, for everyoneThis post announces the public beta of Hex's Magic AI-assist tools, which provide AI assistance for generating, editing, debugging, and documenting SQL and Python. Key improvements include upgrading to GPT-4 for most tasks, implementing streaming responses for longer completions, and introducing 'Magic CTE Explode' to break down nested queries and 'Magic Cell Title' for fun. The post also emphasizes privacy and security measures, noting that Hex does not train models on customer data and does not send underlying customer data to models, but does use schema information and project code as context. It also mentions the team is looking for AI engineers.
  33. Introducing: Hex MagicThis post introduces Hex Magic, a private beta suite of AI tools designed to augment data practitioners. It highlights AI's ability to assist with writing SQL and Python code, explaining and documenting code, and making recommendations based on database schemas, past operations, and project execution graphs. The core contribution is the introduction of AI-powered code generation and assistance within the Hex notebook environment, emphasizing human-in-the-loop interaction and partnership.