Workflow Scheduling

4/10/2025

This post details Mediaset's migration from a legacy Airflow and AWS EMR/Glue/Athena/Redshift stack to Temporal and Bauplan. It highlights the benefits of using Temporal for orchestration, specifically its durable, replayable workflows with automatic retries and built-in scheduling, which replaced Airflow and improved data freshness from 1 hour to 5 minutes. The post contrasts Temporal's code-centric approach with Airflow's YAML-based debugging and emphasizes Temporal's reliability and ease of use for complex flows.

11/14/2024

This post predicts that workflow-driven microservices will become a more popular architecture in financial services in 2025. It highlights the challenges of microservices such as transactional integrity (lack of XA transactions, need for Saga pattern), difficulty in understanding and managing business processes (microservice-to-microservice calls, central choreography, event-driven architectures), mixing business processing with system error handling, challenging observability, and reliability issues. The post suggests that code-first workflow technologies like Temporal can solve many of these problems by making mission-critical services more reliable and enabling rapid development of complex systems and AI applications.

10/30/2024

This post details how Temporal Workflows can be used to manage complex deployment processes, including multi-repository builds, blue-green deployments, and manual approval steps. It illustrates how Temporal's durable execution model simplifies these scenarios by providing built-in retries, state management, and fault tolerance, reducing the need for custom "systems" to handle these complexities. The post uses examples from GitHub Actions and discusses how Temporal can abstract away much of the underlying infrastructure and coordination.

4/24/2024

This post introduces the Workflow History Export feature on Temporal Cloud, enabling users to export closed workflow histories to AWS S3 for analysis. It details the process of transforming Protocol Buffer data to Parquet for easier analysis, setting up ETL pipelines, and provides sample SQL queries for analyzing workflow and activity execution frequency and duration. It also discusses building dashboards for visualization and outlines current limitations of the export feature.

12/2/2022

This post details the technical decisions and challenges in building Temporal's new Scheduler feature as a Workflow. It covers the rationale for choosing a Workflow-based implementation over other options, the decision to house it within Temporal Server, and the engineering challenges related to Namespace management, hiding internal Workflows using search attributes ('Namespace division'), and the implementation of signals and queries for state management, including a workaround for synchronous updates.