TechPrufer
Get started
Temporal
Temporal logoTemporal · Feature trail

Workflow History Export

1 post · Mar 2024 – Apr 2024

What this trail covers

Temporal Cloud allows users to export closed workflow histories to AWS S3 for long-term storage, compliance, auditing, and analysis. This enables feeding workflow history data into data warehouses and building custom dashboards for operational insights, cost inference, and implementation refinement. The export process involves converting Protocol Buffer (proto) formatted history data to Parquet for easier analysis, with considerations for data transformation, schema evolution, and deduplication.

All Temporal trails
  1. Workflow History Export Insights on Temporal CloudThis post details the process of exporting closed workflow histories from Temporal Cloud to AWS S3. It covers transforming the exported Protocol Buffer (proto) format into Parquet for easier analysis, including flattening nested structures and including `workflowID` and `runID` for relationship preservation. It also provides example SQL queries for analyzing workflow and activity frequency, average execution times, and longest-running activities. Limitations such as 'at least once' export guarantee, exclusion of encrypted payloads, and schema evolution are discussed. The post also mentions potential future analyses like inferring cost, analyzing retry policies, task queue usage, SDK versions, worker version changes, and failure messages.
  2. Introducing Workflow History ExportThis post announces the public preview availability of Workflow History Export in Temporal Cloud, allowing users to export closed workflow histories to AWS S3. It details the setup process involving an AWS S3 bucket in the same region as the namespace and granting Temporal permission via AWS CloudFormation. Exports occur hourly. The post also mentions the cost associated with this feature (each exported workflow accrues a single action) and provides links to documentation, pricing, and the changelog.