Data Transformation Workflows

4 posts · May 2021 – May 2026

What this trail covers

Hex has evolved its data transformation capabilities by providing a dedicated workspace for analytics engineers. This includes streamlining the development of data transformations through features like input parameters for automated common checks, a non-linear SQL notebook format for iterative development and debugging, and integrated chart cells for visualizing transformed data as work progresses. The platform aims to keep users in flow by consolidating queries, models, and visualizations in one place, and emphasizes making transformation work visible and collaborative through version control, code review, and documentation.

All Hex trails

May 2026A Guide to Data TransformationThis post details the challenges and best practices for data transformation, focusing on maintaining trustworthiness as teams grow. It outlines core transformation techniques in SQL and Python (cleaning, reshaping, joining, aggregating, CTEs), discusses the evolving ownership of transformation work by analytics engineers, and highlights common issues like metric definition drift, lack of documentation, and the 'bus factor of one'. It emphasizes making transformation work visible and collaborative through version control, code review, and documentation, and contrasts the limitations of traditional BI tools and semantic layers in addressing these issues.
May 2026Ad Hoc Queries: Definition, Benefits & Best PracticesThis post details the challenges and best practices surrounding ad hoc queries, emphasizing the need for governed context and semantic modeling to ensure consistency and prevent knowledge loss. It highlights how Hex's platform supports these needs by enabling natural-language analysis that generates SQL, integrating with semantic modeling, and providing tools for governed self-service analytics. The post contrasts the limitations of traditional dashboards with the flexibility of ad hoc analysis, advocating for a hybrid approach where governance is embedded into the ad hoc workflow.
May 2026Data Orchestration: How It Works And Where It StopsThis post details the limitations of data orchestration alone in solving the 'last mile' problem, where data is technically available but doesn't drive decisions. It explains that while orchestration handles scheduling, execution, and dependency management for data collection and transformation, it doesn't address issues like metric consistency, trust in dashboards, or business users bypassing governed data for external tools. The post advocates for extending governance into the analytics layer through centralized metric definitions, consistent access controls, and semantic lineage that traces metric calculations and changes. It highlights how combining orchestrated data with a governed analytics layer reduces manual handoffs, enables effective self-serve analytics, and shifts data teams towards more strategic work.
May 2026ETL Pipelines: Architecture, Examples, and Where They BreakThis post details the architecture, examples, and failure points of ETL pipelines. It explains the Extract, Transform, Load stages, contrasts ETL with ELT and broader data pipelines, and provides five detailed examples: e-commerce data consolidation, financial reporting and compliance, ML feature engineering, cross-team self-serve analytics, and real-time inventory/operations. It also discusses processing modes (batch, streaming, CDC), common ETL tools, and how AI is changing ETL. The post highlights the importance of downstream usability and the challenges of data quality and accessibility.
Feb 20267 Data Analysis Techniques: A Practical GuideThis post details seven common data analysis techniques (regression, Monte Carlo simulation, factor analysis, cohort analysis, cluster analysis, time series analysis, and sentiment analysis) and how an agent can apply them, emphasizing the importance of choosing the right technique for the question at hand. It also covers quantitative vs. qualitative data and a framework for conducting analysis end-to-end.
Advertisement
Dec 2025What are data pipelines? How they work & how to chooseThis post introduces and explains the concept of data pipelines, detailing their components (connectors, transformations, orchestration, collaboration), types (batch, streaming, hybrid), and criteria for choosing the right one (scalability, governance, collaboration). It highlights how Hex facilitates these aspects within its platform, particularly for data transformation workflows.
Dec 2024How we renovated our data warehouse without interruptionThis post details the implementation of a blue-green deployment strategy for renovating the data warehouse, focusing on maintaining production access during the renovation process. It outlines the steps for setting up a parallel development environment, managing code branches, and configuring orchestration for the new environment, emphasizing the importance of clear goals and iterative development. The technical details include creating a separate working database and schemas, establishing a 'working-main' git branch for development, and configuring dbt Cloud environments and jobs to build and test changes in isolation from the production environment. It also touches upon the use of Snowflake's `swap` syntax as a potential alternative for blue-green deployments.
Aug 2024How Recursion’s data team accelerates drug discoveryThis post details how Recursion's data team adopted Hex to transform their drug discovery workflow from a horizontal pipeline to a scalable loop. They integrated Hex with their BioHive supercomputer (Prefect/GKE) and BigQuery to decouple data processing from storage and last-mile delivery. Hex is used for prototyping, combining, and visualizing messy data outputs from the supercomputer, enabling faster iteration and democratized access to insights. This shift allows for more holistic decision-making by providing interactive data apps and reports to leadership, reducing reliance on expensive data pipelines.
Jul 2023Using SQL for Data AnalysisThis post details how to leverage SQL for data analysis, covering foundational commands (SELECT, WHERE, GROUP BY, HAVING, ORDER BY, JOIN, Subquery, CASE), more complex techniques (window functions, PARTITION_BY, CTEs, Correlated Subqueries, PIVOT), and performance optimization. It emphasizes the efficiency of performing data analysis directly within the database using SQL, reducing the need to pull large datasets into other languages like Python or R. It also touches upon the concept of stored procedures for implementing data analyses within databases.
Apr 2022Hex for Analytics Engineers: Data TransformationThis post introduces Hex as a platform for analytics engineers to develop data transformations. It details how input parameters can automate common data exploration tasks, how the SQL notebook format facilitates non-linear development and debugging of transformations, and how integrated chart cells aid in visualizing data during the transformation process. The post emphasizes the benefits of having all work in one place for improved efficiency and communication.
May 2021Hex Latest and Greatest: May 2021 - BlogThis post introduces several new features to Hex: a SQL IDE with query parameterization, query caching, and a schema browser/typeahead; a beta Chart Cell for no-code visualization; @mentions and app comments for enhanced collaboration; and an improved scheduling system with a run history. These features aim to streamline data exploration, visualization, and collaboration within the Hex platform.