Hex has evolved its agent evaluation capabilities with the introduction of Kimi K2.7, an open-weight model that demonstrates comparable intelligence to frontier models like Opus 4.7 at a significantly lower cost. The post details the evaluation of Kimi K2.7 against Opus 4.7 across various analytical tasks, highlighting its strengths in semantically modeled and analytically hard questions, while noting areas for improvement in visualization and contextually hard tasks. This integration represents a strategic bet on open-weight models, enabling new tradeoffs between cost, speed, and intelligence for data analytics.
8/4/2026
Introduced Evals, a new system for testing and measuring Hex Agent performance. Evals leverage an LLM-as-judge to evaluate agent reasoning and tool usage, not just final answers. The system is integrated with the Hex CLI and supports version-controlled test cases and context previews for safe testing of changes. Evals enable scheduled checks for regressions and configuration sweeps across different LLMs.
7/29/2026
This post extends the utility of the Evals system by framing its output as a tool for building business trust and driving adoption, rather than solely for internal debugging. It emphasizes the strategic communication of eval results, including sample questions and domains, to demonstrate the agent's reliability and the data team's work in organizing context. It also connects eval scores to adoption metrics, suggesting a new way to measure the impact of data team efforts.
7/14/2026
Introduced a credit model for Hex Agents, including monthly credit grants per user seat (Professional, Team, Enterprise, Explorer) and pooled add-on credits with auto top-ups. Implemented an effort-based consumption system that dynamically selects LLM models based on task complexity, context, and resource requirements, aiming for cost efficiency. Enhanced visibility by allowing users to view credit balances and per-prompt costs, and providing admins with historical usage logs, spend limit settings, and credit usage data within the Context Studio. This enables better budgeting, ROI calculation, and informed decisions on context curation and model selection.
7/8/2026
This post introduces Kimi K2.7 as a new, cost-effective open-weight model for analytics within Hex. It details the evaluation methodology comparing Kimi K2.7 against Opus 4.7, presenting quantitative results on runtime, cost per task, and success rates across different evaluation sets (semantically modeled, semantically unmodeled, core visualization, analytically hard, contextually hard). It discusses the tradeoffs of open-weight models, specifically Kimi K2.7's tendency for more turns and validation in analytically hard tasks, and its limitations in context window size and visualization efficiency. The post also touches on the future potential of hosting and configuring open-weight models.
6/9/2026
This post details the creation and application of new, more challenging evaluation benchmarks ('Frontier' eval set) for LLMs in data analysis contexts. It quantifies the performance improvements of Claude Fable 5 on these benchmarks, highlighting its superior analytical reasoning, ability to handle complex, long-horizon tasks, and better adherence to the 'golden workflow' of data analysis. The post also discusses qualitative improvements in how the model frames assumptions and communicates findings, contrasting it with previous models.
5/22/2026
This post details the evolution of the 'Evals' system into a comprehensive lab bench called 'The Shoebox' for agent observability and evaluation. It introduces a synthetic business, 'Shorelane Commerce,' with realistic data and complexities to serve as a benchmark for agent evaluations. The post elaborates on the infrastructure for running evaluations as pairwise experiments, the use of rubrics and ground truths, and the creation of hypothesis objective rubrics for targeted evaluations. It also discusses the challenges and maintenance burden associated with such a system.
Unknown date
This post introduces Context Studio, a system for observing, testing, and deploying AI agents. It details capabilities for managing context (workspace rules, domain-specific guides, semantic models), observing agent performance (tracking trends, surfacing agent confusion and context gaps via the Review Agent), and reviewing/publishing updates (triage, suggestions, CLI integration). It also highlights how context curation fits into modern data stacks and integrates with tools like dbt MetricFlow.