BlogsTemporalDurable Execution

Durable Execution

Durable Execution

108
posts
2021–2026

Temporal Workflows are used to build resilient platform control planes by mapping resources to long-running Entity Workflows. This pattern leverages Updates, Signals, and Queries for message handling, serializes side effects in a main loop, and uses Continue-as-New for history rotation. It supports complex lifecycles, error handling, compensation, and integrates with external APIs via Activities. This post details how a durable egress-only connector for cloud-to-edge messaging was built using Temporal. The saga pattern provides a robust, modern solution for handling data consistency challenges in microservices by coordinating distributed transactions through sequences of local transactions with compensating actions. It can be implemented via choreography or orchestration and is beneficial for e-commerce, finance, and IoT systems. Best practices include designing for failure, ensuring idempotency, and defining compensation actions. Temporal's platform has been instrumental in enabling organizations to adopt the saga pattern effectively.

2026

Durable, flexible multi-agent systems

8/6/2026

This post demonstrates how Temporal's Durable Execution layer can be used to build robust multi-agent systems that can handle varying latencies and human-in-the-loop interactions. It showcases how Temporal Workflows can be used to orchestrate agents built with different frameworks (ADK and LangGraph) and how human interactions can be modeled as durable signals, surviving worker crashes and infrastructure changes. The post also highlights the use of Continue-as-New for managing long-running driver workflows and the importance of separate task queues for agent reasoning and driver activities.

How Temporal powers workflows at HeyGen

8/6/2026

HeyGen details their implementation of Temporal for video generation, including modeling video generation as Temporal Workflows and individual jobs as Activities. They describe how they handle scene orchestration within a single workflow using concurrency limits and selective use of child workflows for isolation. The post also covers their approach to customer-level fairness through an admission scheduler and Temporal's task priority, and the development of a shared execution layer for cross-cutting concerns like identity propagation, error classification, metrics, and search attributes. They also discuss reusing workflow modules as inlined components versus child workflows.

I spent two months screen-sharing to ship one integration. So I rebuilt it on Temporal.

8/6/2026

This post details the construction of an egress-only, durable proxy application for cloud-to-edge messaging using Temporal. It replaces a complex, custom-built solution involving message queues, manual retry logic, and extensive testing with a simpler, more reliable Temporal-based architecture. The new system utilizes Temporal Workflows to manage configurations and an outbound connection from the edge device to Temporal for communication, eliminating the need for inbound firewall rules and manual debugging sessions. The author highlights the benefits of Temporal's durable execution, out-of-the-box reliability, and visibility in solving the challenges of integrating with firewalled edge devices.

The bot that lied to cricket fans with a straight face: Temporal x San Francisco Unicorns

8/4/2026

This post details the development of a WhatsApp bot for San Francisco Unicorns fans, showcasing the use of Temporal Workflows for managing asynchronous tasks like match reminders and live-score polling. It highlights the `conversationTurnWorkflow`, `matchReminderWorkflow`, `matchScoreSubscriptionWorkflow`, and `postMatchSummaryWorkflow` for handling user messages, scheduling reminders, polling scores with Continue-as-New for history management, and sending post-match summaries. The post also discusses challenges with WhatsApp's 24-hour messaging window, webhook verification, and duplicate message handling. A critical failure scenario where the bot provided stale scores due to an incorrect budget limit implementation is described, emphasizing Temporal's role in ensuring workflow continuity and recovery.

I had two weeks, an 8-foot keyboard, and a Wordle Workflow

7/21/2026

This post details the development of a Wordle game where the Temporal Workflow itself acts as the game state. It highlights how the Workflow manages game logic, state, and completion conditions, with the UI serving as a thin layer. The post also discusses the practical challenges of running such a demo at a conference booth, including direct laptop integration with a giant keyboard, real-time UI updates, and making Temporal's event history visible on a holographic fan display. It emphasizes the reliability provided by Temporal's durable execution for managing state in a chaotic, live environment.

LangGraph in production: Temporal's LangGraph Plugin adds Durable Execution

7/16/2026

Introduces a LangGraph integration for Temporal Python SDK, allowing LangGraph agents to be executed as Temporal Workflows. This provides durable execution, automatic recovery from failures, and cost-effective human-in-the-loop interactions by leveraging Temporal's signals and timers. Also announces LangSmith integrations for Python and TypeScript to provide end-to-end tracing of durable agent runs.

Why a context engine needs Durable Execution: How Unblocked rebuilt its ingestion backbone on Temporal

7/14/2026

Unblocked adopted Temporal for its AI context engine's ingestion backbone, migrating from a DIY actor model. The implementation leveraged Temporal's polyglot SDK support and Durable Execution primitive, collapsing orchestration code significantly. The system now runs ~1 million Temporal Workflows daily, self-hosted on Kubernetes, with benefits including deployment-free workflows, elimination of deploy-time database CPU spikes, unified handling of user-triggered and cron flows, and built-in observability.

Why I needed Durable Execution to read a toy manual

7/13/2026

This post details the architecture and implementation of Toku Solutions, a pipeline for translating Japanese toy manuals into editable static sites. It highlights the use of Temporal Workflows and Activities for managing the pipeline, including fan-out OCR by page, retry policies for translation APIs, heartbeating for long-running site generation, and the use of LLMs (Gemini) within Activities for AI cleanup. The post emphasizes the importance of Pydantic validation for LLM output to maintain deterministic Workflow execution and the benefits of Durable Execution for handling failures and partial progress.

Why teams run Temporal workers with Encore

7/9/2026

This post describes how Encore integrates with Temporal to provision and manage the infrastructure for Temporal workers. It explains how Encore can automatically provision resources like databases and Temporal credentials, and how Temporal workers can be run as Encore services. The post highlights the benefits of combining Temporal's durable execution with Encore's infrastructure automation for building resilient backends and for AI agents.

How Rapidflare built a million+ document ingestion pipeline for Agents on Temporal

7/8/2026

This post details Rapidflare's implementation of a million+ document ingestion pipeline using Temporal. Key technical contributions include: a sliding window mechanism for managing a large number of Child Workflows to control concurrency and maintain worker utilization; the use of Continue-as-New to manage large Workflow histories by checkpointing state at page boundaries; the implementation of Activity Heartbeats with milestone reporting for robust progress tracking during long-running crawl Activities; the use of Cloud Storage as a data bus for staging large document artifacts, overcoming Temporal's payload limits and enabling shared state across distributed workers; deterministic Child Workflow IDs to prevent duplicate processing; and the `ParentClosePolicy.ABANDON` to ensure children survive parent restarts. The post also highlights the use of separate Task Queues for different processing stages (Ingestion, Enrichment, Status sync) to provide isolation and prevent starvation.

How Sherlocks AI uses Temporal to orchestrate AI agents for incident resolution

7/7/2026

Sherlocks AI details their use of Temporal to orchestrate AI agents for incident resolution. They describe four distinct workflow patterns (Knowledge Graph construction, Infrastructure scanning, AI agent investigations, Event ingestion) that are all managed by Temporal. The post highlights Temporal's Durable Execution, Retry Policies, Scheduling, Signals for human-in-the-loop interaction, and Observability as key enablers for their platform. They also discuss their architectural choices, including per-tenant Workflows for isolation and the use of a self-hosted Codec Server for data encryption.

How we built it: Running Replay on Temporal

6/24/2026

This post details the architecture and implementation of a conference registration system built on Temporal. It showcases how Workflows were used to manage the entire lifecycle of an order and individual tickets, from initial purchase through to event check-in. Key technical aspects include using Workflows for long-running, stateful processes, employing Activities for external integrations (payment processing, email, database), implementing idempotency for critical operations like payment charging, and leveraging Temporal's event history for debugging and auditing. The post also discusses the decision to use a separate Ticket Workflow per ticket to manage individual attendee lifecycles and avoid large event histories.

Stop failing on the path to production: A better way for agentic platforms

6/23/2026

This post positions Temporal's Durable Execution as a foundational harness for building agentic platforms. It outlines the requirements for successful agent architectures (memory, resilience, programming model, auditability, visibility, unlimited runtime, data typing, LLM/tool orchestration, customization, full orchestration control) and maps them to Temporal's core primitives (Workflows, Activities, Signals, Queries, Updates). It emphasizes the importance of secured data and deployed agents over desktop agents, and provides a reference sample harness and links to related resources for building agentic platforms with Temporal.

How Kelet AI built a durable agent that debugs durable agents.

6/18/2026

This post details how Kelet AI built a durable agent for diagnosing AI quality failures using Temporal. It describes a four-level Temporal Workflow hierarchy (Session, Signal, Agent Aggregation, Investigate Issue) to handle long-running state, durable execution, event-driven coordination, and cross-workflow gating. The Session Workflow waits for a silence window before analysis and rebuilds state from the database on startup. Signal Workflows run in parallel for enrichment and interrogation. Agent Aggregation Workflows collect hypotheses across sessions, and Investigate Issue Workflows reason over failure clusters to produce root causes and prompt patches. The post also covers a Temporal plugin for integrating Kelet's monitoring capabilities, propagating session attributes via headers for seamless tracing and debugging of Temporal-based agents.

How Monk migrated 100+ workflows from Inngest to Temporal

6/16/2026

This post details a practical, incremental migration strategy for moving over 100 workflows from Inngest to Temporal. It outlines a four-step process (Characterize, Scaffold, Cut over, Remove) using feature flags and shared business logic to ensure safety and reversibility. It provides a detailed mapping of Inngest concepts (cron triggers, concurrency, retries, step.run) to their Temporal equivalents (Schedule, Workflow, Activity, Schedule overlap, Activity retry policies), highlighting critical differences in retry counting and the need to move non-deterministic code into Activities during migration. A specific example of migrating a scheduled drift check workflow is provided.

Coordinate Access to Shared Resources with a Distributed Lock Built on Temporal Workflows

6/11/2026

Introduces a validated pattern for implementing a distributed lock using Temporal Workflows. Each lock/permit is a short-lived child Workflow, leveraging Temporal's atomic workflow ID uniqueness for acquisition and lease timeouts for orphan recovery. This pattern avoids external dependencies like databases or rate limiters.

Diving into the AI iceberg: What lies beneath your AI agents

6/10/2026

This post details how Temporal orchestrates a multi-agent AI research workflow, highlighting its role in coordination, human-in-the-loop interactions, parallel execution of search activities, automatic retries for transient failures, and ensuring durable execution for long-running processes. It provides a simplified code example of the workflow structure and emphasizes the value of Temporal in making such complex AI applications resilient and observable.

How Coverwatch uses Temporal to orchestrate AI-powered insurance workflows

6/9/2026

This post details how Coverwatch uses Temporal to orchestrate AI-powered insurance workflows, combining AI agents, human review, carrier communication, and operational follow-up. It highlights the use of Temporal's durable execution guarantees with agentic automation through Pydantic-AI integration, and how Temporal Signals are used to integrate human review into the durable flow. The post also emphasizes the value of workflow history for debugging and iteration by both engineers and AI coding agents.

How to build deep research agents using Temporal and Braintrust

6/3/2026

This post details the implementation of a multi-agent system for deep research agents using Temporal Workflows and Activities. It showcases how Temporal's Durable Execution addresses common failure modes in LLM-based pipelines, such as API timeouts and partial failures, by mapping agents to Activities with appropriate retry semantics and timeout configurations. The post also describes the use of Pydantic models for structured data contracts between agents and the integration of Braintrust for end-to-end observability and evaluation of the research pipeline.

How to build a resilient platform control plane with Temporal

6/1/2026

This post introduces the Entity Workflow pattern for building resilient platform control planes. It details how to map resources to long-running Workflows, manage state, and handle external interactions using Temporal's message passing primitives (Updates, Signals, Queries). The post also covers the main loop for serializing side effects, error handling, compensation, and the use of Continue-as-New for history management, with Python SDK examples.

How groundcover built a durable alert dispatch system with Temporal

5/29/2026

This post details how groundcover built a durable alert dispatch system using Temporal's durable execution, SignalWithStartWorkflow, and per-alert Workflows to handle high-throughput fan-out and maintain ordering within alert streams. The system uses a two-level workflow architecture with an orchestrator and per-alert Workflows, signal batching for efficiency, and visibility API for backpressure management. State is managed in-memory within Workflows with persistence to PostgreSQL, leveraging Temporal's replay for durability. Retry policies are used for Activities, and notification delivery Activities handle external rate limits and retries.

What does it take to build a customer support experience your users won't hate? Ask Bitovi.

5/19/2026

This post details how Bitovi built a stateful AI agent for customer support using Temporal Workflows and Activities. The ReAct agent loop is implemented as a Temporal Workflow, with LLM inference and memory extraction handled by Activities. The post emphasizes the benefits of Temporal's Durable Execution for handling long-running, stateful conversations and integrating external memory systems like AWS Bedrock AgentCore Memory. It highlights the isolation of nondeterministic LLM inference within Activities and the use of structured output with validation and retries for reliability. The post also provides a code snippet for a `ThoughtActivity` and links to the full implementation on GitHub.

From FileUploadWorkflow to creative OS: How Layer scaled on Temporal

4/28/2026

Layer.ai details their evolution from a simple file upload workflow to a complex AI asset generation platform powered by Temporal. They've built a visual pipeline builder where each node is a Temporal child workflow, using `workflow.wait_condition()` and a shared `all_values` dictionary as a durable data bus. They also highlight how their `RunInferenceWorkflow` supports multiple modalities (image, video, 3D, audio) without requiring redesigns, adapting to longer processing times and larger files. The post also touches on the integration of LLM agents to invoke Layer workflows autonomously.

From agent zoo to agent orchestra: The benefits of Temporal as your enterprise agentic control plane

4/23/2026

This post details how Temporal can be used as an enterprise agentic control plane, extending the concept of Durable Execution to AI agents. It categorizes agents into Conversational, Event Stream Processors, Ambient, and Agent-to-Agent interactions, and explains how Temporal Workflows provide essential features like guardrails, human-machine interaction, versionability, durability, cost management, visibility, and auditability for these agent types. It introduces Temporal Nexus as a mechanism for coordinating agents across namespaces and discusses Workflow Versioning for managing changes in AI agent logic. The post also touches upon operational realities like LLM cost management and retry policies.

How XY builds an AI agent orchestration platform for healthcare with Temporal

4/23/2026

This post introduces XY's AI agent orchestration platform for healthcare, built on Temporal. Key contributions include: a YAML-based DSL interpreted by a single generic `DSLWorkflow` class, enabling declarative workflow definition; an AI Planner Agent that translates natural language into executable YAML and generates activities; a dynamic Activity Factory for runtime assembly of customer-specific integrations; and a deferred continue-as-new mechanism for handling large-scale batch processing. The post also highlights the benefits of Temporal's vocabulary for team alignment and its role in managing data sensitivity and human-in-the-loop processes.

What agentic AI borrowed from microservices (and made worse)

4/23/2026

This post draws parallels between microservices architectures and agentic AI, highlighting how concepts like decomposition, orchestration, event-driven architectures, and state management are reappearing in the AI space. It specifically discusses how Temporal's durable execution model addresses the challenges of long-running AI workflows, short-term memory, and the need for event-driven architectures, positioning Temporal as a solution for building reliable and scalable AI agent systems.

Introducing Temporal and agentic sandboxes: The OpenAI agents SDK

4/16/2026

This post details the integration of Temporal's Durable Execution with OpenAI's Agents SDK to provide fault-tolerant execution for sandbox agents. It introduces an `AgentWorkflow` that wraps the `SandboxAgent` to persist state and idle efficiently, a `SessionManagerWorkflow` for orchestrating agent sessions, and a TUI for client interaction. Key technical contributions include the use of `workflow.wait_condition` for durable idling, forking workflows across different sandbox backends by snapshotting state, and wrapping sandbox operations as Temporal activities via `temporal_sandbox_client`.

AI reliability is a decade-old problem. And we’re still only solving half of it

4/1/2026

This post argues that the current focus on AI model reliability (accuracy, bias, hallucination) is insufficient for production AI agents. It emphasizes the critical need for system-level reliability, specifically Durable Execution, to handle failures in long-running, multi-step workflows. The author contrasts the limitations of current AI agents in recovering from failures with the capabilities of Temporal's Durable Execution, which provides checkpointing and resumption for distributed processes. The post uses examples like the Google Antigravity incident and METR's research on time horizons to illustrate the problem and positions Temporal's infrastructure as a solution for AI agent reliability.

Why your platform control plane belongs on Temporal

3/31/2026

This post introduces the application of Temporal's Durable Execution to platform control planes. It explains how Workflows can manage the entire lifecycle of infrastructure (creation, monitoring, upgrading, modification, decommissioning), eliminating orphaned states and manual cleanup. It highlights the use of Temporal for scheduled and reliable automation, replacing fragile cron jobs with resilient Workflows that offer out-of-the-box retries, timeouts, and deep observability. The post also details how Temporal enables true self-service by supporting long-running execution and human-in-the-loop patterns via Queries and Signals, allowing for safe exposure of complex infrastructure operations to internal customers.

How I shipped a Temporal-powered product in three weeks, one year out of college

3/26/2026

This post details the author's experience building a property management system using Temporal Cloud, focusing on the implementation of a maintenance request workflow. It highlights the use of Parent and Child Workflows, Signals for human interaction, and Activities with retry policies for integrating with unreliable external systems. The author also reflects on an anti-pattern in their initial polling loop and provides a more idiomatic solution using server-side Activity retries.

Building AI agents that overcome the complexity cliff

3/10/2026

This post introduces a taxonomy of agent capability levels (Reflexive, Conversational, Orchestrated, Coordinated, Autonomous) and identifies the "complexity cliff" where traditional frameworks break down. It argues that Durable Execution is required above this cliff to handle failures and enable efficient iteration for complex AI agents. The post explains how Durable Execution's automatic state persistence and replay capability address these challenges, contrasting it with less effective standard approaches like cron restarts or basic checkpointing. It highlights Temporal's ability to facilitate fast and efficient parallel experiments by forking execution history, enabling testing of prompt variants without re-executing completed work.

Temporal drives demand for Durable Execution

2/17/2026

This post announces a significant funding round and highlights the growing demand for Durable Execution, particularly driven by AI workloads. It quantifies the platform's growth with metrics like revenue increase, weekly active usage, and installs. It also mentions ongoing investment in AI-native features and developer experience.

What will AI do to your career?

1/27/2026

This post positions Temporal's Durable Execution as critical infrastructure for the AI era, emphasizing its role in managing complex, long-running AI workflows and the emerging agentic stack. It highlights new integrations with OpenAI Agents SDK, Pydantic AI, Langfuse, and Vercel AI SDK, and introduces a new Braintrust integration for AI agent visibility. The post argues that Temporal's framework-agnostic approach to orchestration, durability, observability, and fault tolerance makes it essential for scaling AI systems from small experiments to millions of agents.

Trusting AI agents: A reinsurance case study

1/22/2026

This post details the implementation of a multi-agent AI system for reinsurance data processing using Temporal Workflows and Activities. It highlights the use of Temporal Signals for human-in-the-loop confirmation of tool executions and agent completions, and Temporal Workflows as a central router (Bridge Workflow) and inter-agent data store. The post also discusses principles for building AI agents, including designing for human-in-the-loop, using modular sub-agents, and keeping tool arguments simple, all within the context of Temporal's durable execution guarantees.

Building durable agents with Temporal and AI SDK by Vercel

1/20/2026

This post details the integration of Temporal's Durable Execution with the AI SDK by Vercel. It introduces a plugin (`@temporalio/ai-sdk`) that wraps AI SDK calls (like `generateText`) in Temporal Activities, providing automatic retries and state persistence for LLM interactions. It also explains how to implement tool execution within Temporal Workflows by proxying Activities, ensuring that external API calls made by AI agents are also durable. A real-world example of a daily news briefing agent demonstrates how to combine these durable LLM calls and Activity executions for complex, multi-phase AI workflows.

Building observable AI agents: Temporal now integrates with Braintrust

1/20/2026

This post details the integration of Temporal with Braintrust to provide observability for AI agents. It introduces code examples for wrapping OpenAI clients with Braintrust's `wrap_openai` and integrating the `BraintrustPlugin` into Temporal Workers and Clients. It also demonstrates how to use `braintrust.load_prompt` to manage prompts externally from Temporal Workflows.

The hero’s journey to AI durability with Temporal

1/14/2026

This post details how Temporal's Durable Execution model addresses the challenges of building reliable AI features in production. It explains how Workflows and Activities can be used to handle multi-step AI flows, ensuring idempotency for model calls, managing human-in-the-loop processes with Signals and Queries, and orchestrating tool execution within AI agents. Key patterns discussed include using Activities for external work (model calls, tool execution) and Workflows for state management and orchestration, ensuring that progress is not lost upon worker restarts or failures. It also emphasizes the use of Temporal timers for durable waits within Workflows and the integration with MCP for tool execution.

2025

The journey to shippable AI systems: Patterns that work

11/18/2025

This post details five patterns for building shippable AI systems: guarded tool calls (timeouts, retries, circuit breakers, fallbacks, input/output persistence, cost metadata), idempotent side-effects (write-once semantics, compensating actions), human checkpoints (approval steps, handling reviewer timeouts), deterministic plans with versioned prompts (orchestration stabilization, prompt/model version recording), and cost/policy budgets (token/spend tracking, graceful degradation). It argues that Durable Execution is the backbone for these patterns, providing state, retries, visibility, and long-running task management for AI workflows.

Of course you can build dynamic AI agents with Temporal

11/12/2025

This post clarifies that Temporal's deterministic Workflow requirement does not preclude building dynamic AI agents. It explains that the separation of deterministic Workflows (orchestration) and non-deterministic Activities (LLM calls, tool invocations) is key. The post details how Temporal's Durable Execution allows AI agents to make runtime decisions based on LLM outcomes, ensuring that if a failure occurs, the agent can resume from its last consistent state without re-executing LLM calls or diverging from its original path. It provides code examples and architectural diagrams to illustrate this concept, highlighting how Temporal enables non-deterministic agents with durable execution.

Here's how to build durable AI agents with Pydantic and Temporal

11/6/2025

Introduces native Temporal support in Pydantic AI, allowing AI agents to leverage Temporal's durable execution for fault tolerance, long-running interactions, and state preservation. This integration automatically wraps non-deterministic agent work (LLM calls, tool executions) as Temporal Activities with built-in retries, while the agent's coordination logic runs as a deterministic Temporal Workflow. Demonstrates this with a two-agent system for a Slack dinner recommendation bot, showcasing orchestration, handling asynchronous messages with Signals, coordinating API calls as Activities, and implementing human-in-the-loop approval flows.

How to think about agentic solutions for the enterprise

10/21/2025

This post highlights the critical gap between AI experimentation and production-ready systems, emphasizing the need for Durable Execution as the missing backbone for enterprise AI solutions. It details how Durable Execution addresses the challenges of running AI reliably over extended periods, handling failures gracefully, maintaining visibility, and scaling predictably. The post contrasts this with current AI tooling that focuses on iteration speed rather than production durability and provides examples of companies that have successfully adopted Durable Execution for their AI initiatives.

Building a persistent conversational AI chatbot with Temporal

10/14/2025

This post details the technical implementation of a persistent, infinitely scalable chatbot using Temporal Workflows. It outlines the functional and non-functional requirements for such a system, contrasts it with traditional stateful chatbot architectures, and presents a Temporal-based design where each conversation is a Workflow. Key technical aspects include using Signals for message processing, Activities for interacting with LLMs and external tools, and leveraging Temporal's durable execution for resilience against failures, restarts, and scaling events. The post also discusses the benefits of this approach, such as simplified state management, infinite scalability, resilient workflows, maintainable design, rich context management, transparent state inspection, and robust error handling.

Individual contributors vs. decision makers: Same systems, different realities

9/23/2025

This post analyzes the State of Development 2025 survey data to highlight the differing perspectives of individual contributors (ICs) and decision makers (DMs) on system realities, particularly concerning reliability and operational overhead. It demonstrates how Temporal's Durable Execution model addresses pain points for both groups by translating IC-focused issues like latency and debuggability into DM-focused benefits such as reduced business exposure, security, and cost efficiency. The post emphasizes that Temporal's core offering of survivability and automated retries directly maps to the shared goals of enhanced reliability, increased automation, and reduced operational costs.

How Cargo engineers a smarter go-to-market process with Temporal

8/27/2025

This post details how Cargo uses Temporal for event-driven GTM workflows, data synchronization (ETL), and AI workflow orchestration. It provides a code example of a Temporal Workflow for orchestrating a "run" and explains the importance of keeping Workflows lightweight and pushing heavy data into Activities. It also discusses the architectural decision to migrate from Postgres to ClickHouse for analytical workloads and the lessons learned regarding the timing of infrastructure investments versus feature shipping.

Building long-running interactive MCP tools with Temporal

8/21/2025

This post details the architecture and implementation of long-running, interactive MCP tools powered by Temporal Workflows. It showcases the Temporal Invoice MCP sample, demonstrating how to expose Workflow interactions (start, status, approve, reject) as MCP tools. Key technical aspects include using `client.start_workflow` to initiate Workflows, `handle.signal` for state changes (e.g., `ApproveInvoice`), and `handle.query` for retrieving Workflow state (`GetInvoiceStatus`). The post highlights the use of `workflow.wait_condition` for human interaction and the benefits of Temporal's durable execution for handling complex, multi-step AI processes.

The fallacy of the graph: Why your next agentic workflow should be code, not a diagram

8/20/2025

This post argues that graph-based orchestration frameworks are fallacious for complex, procedural logic, particularly for agentic applications. It contrasts this with Temporal's Durable Execution model, which allows for normal, procedural code to achieve crashless guarantees. The post details the limitations of graphs in handling dynamic control flow (e.g., LLM-determined tool execution), data management (global key-value stores vs. typed code access), and error handling/compensations (saga pattern complexity in graphs vs. try/except in code). It advocates for using code-based workflows over diagrams for better maintainability, refactoring, and compile-time checks.

The unbreakable web: From fragile scripts to bulletproof Workflows

8/20/2025

This post details the integration of Browserbase, an AI-powered browser automation platform, with Temporal's Durable Execution. It demonstrates how to build "bulletproof" web automation by using Temporal Activities to encapsulate browser interactions managed by Browserbase's Stagehand SDK. Key technical contributions include: - Using semantic selectors (`stagehand.page.act("click the submit button")`) for resilience against website UI changes. - Wrapping browser automation steps in Temporal Activities to leverage Durable Execution for state persistence and automatic retries. - Emphasizing the need for idempotent Activities to safely handle retries, using examples like `initializeBrowser` and `cleanupBrowser`. - Showcasing a resilient search workflow example with atomic Activities and tailored Retry Policies. - Illustrating the combination with screenshots of the Temporal UI and Browserbase console.

Enabling platform engineering with Temporal: 5 practical use cases

8/13/2025

This post details five practical use cases for Temporal in platform engineering: incident response/runbook automation, certificate rotation, infrastructure management, CI/CD pipeline orchestration, and Internal Developer Platform (IDP) automation. It highlights how Temporal Workflows can encode incident response logic, manage certificate rotation with resilience and visibility, automate infrastructure lifecycle management, orchestrate complex CI/CD pipelines with stateful workflows and dynamic rollbacks, and serve as the backbone for IDP automation by encapsulating complex tasks and coordinating across multiple systems. Examples from Datadog, Temporal Cloud, and Netflix are provided to illustrate these applications.

Production-ready agents with the OpenAI Agents SDK + Temporal

7/30/2025

This post details the integration of the OpenAI Agents SDK with Temporal's Durable Execution. It explains how AI agents, defined using primitives like Agents and Handoffs in the OpenAI SDK, can be run as Temporal Workflows. This allows agents to leverage Temporal's capabilities for reliable execution, including handling LLM rate limits, network issues, and application crashes. The integration allows developers to code the happy path for agents, with Temporal managing error handling and ensuring crash-proof execution. The post provides examples of how to define agents and orchestrate them within Temporal Workflows, highlighting the benefits of combining the rapid agent development of the OpenAI SDK with the production-readiness of Temporal.

How Grepsr uses Temporal to deliver scalable and reliable web data

7/24/2025

Grepsr adopted Temporal to orchestrate their web data extraction platform, processing over 600 million records daily. They migrated from a system relying on cron jobs, custom scripts, and message queues to Temporal's code-first Workflows, built-in fault tolerance, native parallelism, and observability features. This resulted in 99% delivery reliability, a 60% reduction in incident resolution time, and faster development cycles for new workflows.

Building production-ready generative AI: How Temporal supercharges Google's Gemini and Veo

7/17/2025

This post details how Temporal's Durable Execution platform is applied to build production-ready generative AI systems, specifically using Google's Gemini and Veo models for video generation. It highlights how Temporal addresses common challenges in agentic systems, such as stateful long-running processes, transient failures, costly re-runs, and visibility/debugging. The post provides a high-level architecture of a video generation system and code snippets demonstrating the use of Temporal Workflows and Activities to orchestrate calls to Gemini for scene generation and Veo for video clip creation. It emphasizes the benefits of structured output from LLMs and Temporal's automatic retry mechanisms for external API calls.

Durable Execution meets AI: Why Temporal is ideal for AI agents & Generative AI Apps

7/10/2025

This post extends the understanding of Temporal's Durable Execution by demonstrating its applicability to AI agents and Generative AI applications. It maps common AI development concepts (chains, LLMs, tools, memory, checkpointing, human-in-the-loop, long-running processes) to Temporal's core features (Workflows, Activities, Signals, Updates, Queries). The post highlights how Temporal's resilience and state management capabilities inherently address the distributed system challenges of AI applications, such as handling numerous remote requests, transient failures, and complex orchestration, without requiring developers to write explicit retry or checkpointing logic.

Durable MCP: Using Temporal to give agentic systems superpowers

7/9/2025

This post introduces the concept of implementing Model Context Protocol (MCP) tools as Temporal Workflows to provide durability, visibility, reliability, and scalability to agentic systems. It demonstrates how to create an MCP tool that executes a Temporal Workflow, which in turn executes a durable Activity. This approach allows agentic systems to leverage Temporal's fault-tolerant execution for tools, ensuring they work every time and can handle enterprise-scale requirements.

From AI hype to durable reality — why agentic flows need distributed-systems discipline

6/11/2025

This post details how to integrate Temporal with AI agentic flows and the Model Context Protocol (MCP). It demonstrates wrapping MCP tool invocations as Temporal Activities, leveraging Temporal's Durable Execution for resilience, automatic retries, timeouts, and state persistence. The post also discusses the application of polyglot workflows for cross-language AI orchestration, the role of Temporal Nexus in unifying AI and business workflows with end-to-end observability, and the use of Signals and Queries to maintain stateful conversations even through failures. It emphasizes how Temporal's features reduce boilerplate code and enhance developer velocity in building AI systems.

How Dust Builds Agentic AI with Temporal Workflows

5/29/2025

This post details how Dust leverages Temporal Workflows as the core orchestration engine for its AI agent platform. It explains how events from various tools (Slack, Notion, GitHub) trigger Temporal Workflows that handle data ingestion, transformation, AI processing, and indexing. It highlights the use of Temporal for Dust's Tracker product, which monitors documents and triggers agent updates. The post emphasizes that Temporal replaces custom logic for retries, timeouts, and state recovery, allowing Dust to focus on product development. It also mentions the scale of operations (10M+ activities/day on Temporal Cloud) and the use of Datadog for observability. A specific example of filtering files within a Temporal Activity for the GitHub connector is provided, illustrating Temporal's retry and resume capabilities.

How Temporal Transformed Workflow Orchestration from Azure and Uber Roots

4/25/2025

This post traces the historical evolution of workflow orchestration and durable execution, from Amazon's SWF and Azure's Durable Task Framework to Uber's Cadence and finally Temporal. It emphasizes Temporal's core concept of 'Durable Execution' as a radical simplification for developers, allowing them to write business logic without explicit failure recovery or state management. The post also highlights the adoption of Temporal by financial institutions due to its resilience and guaranteed outcomes, and discusses Temporal's secure-by-design approach and fully open-source strategy.

How many Activities should I use in my Temporal Workflow?

4/16/2025

This post provides detailed guidance on when to split work into multiple Temporal Activities versus combining them. It categorizes these decisions into 'Necessity,' 'Practicality,' and 'Preference,' offering specific examples for each. Key recommendations include using separate Activities for operations requiring different task queues, retry policies, or timeout policies. It also emphasizes splitting Activities that involve multiple database transactions or state-changing calls to external services, aligning with the principle of one operation per Activity for better maintainability, observability, and reliability.

Temporal: Beyond State Machines for Reliable Distributed Applications

4/11/2025

This post provides a deep dive into Temporal's Durable Execution model, explaining how it functions as an event-sourced state machine. It details the core concepts of Durable Execution, including automatic event logging, persistence, and seamless resumption after failures. The post contrasts Temporal's approach with traditional state machines, highlighting the benefits of writing synchronous-looking code for Workflows and Activities, and how Temporal handles distributed messaging, task queuing, and reliable execution behind the scenes. It also touches upon the challenges of distributed systems like orchestration, state management, retries, timeouts, and visibility, and how Temporal addresses them.

Designing Robust Financial Ledgers with Temporal

3/18/2025

This post details how Temporal's Durable Execution model can be applied to the specific domain of financial ledger design. It elaborates on the challenges of ledger design (high-performance transaction handling, double-entry bookkeeping, flexible metadata) and explains how Temporal's features like stateful workflows, automatic retries, guaranteed execution, scalability, end-to-end visibility, and flexible orchestration address these challenges. It also provides a real-world example of handling reconciliation at scale using Temporal.

Replay ‘25 day two recap: Major product announcements, customer stories, and Durable Execution

3/6/2025

This post details how Temporal's Durable Execution model is being used in production by various companies (Vodafone, noon, Datadog, Mollie, Salesforce, Airwallex, Maersk, Mediaset, Vinted) for diverse use cases like customer premise equipment management, data pipeline orchestration, self-hosting Temporal at scale, microservice orchestration, marketing cloud migration, stabilizing financial platforms, internal developer platforms, data processing, and payment flow management. It also announces several new product launches: Pre-Release Ruby SDK, General Availability of Nexus, enhanced high availability through namespace replication, safer Worker versioning APIs, zero-downtime migration tooling, Temporal Cloud on Google Cloud, and new security and operations tools.

How Layer.ai is upgrading mobile game design

2/21/2025

This post details how Layer.ai uses Temporal's Durable Execution for their AI-driven game asset production workflows. It highlights the use of Temporal to offload heavy tasks like asset generation and file processing, preventing server overload and improving performance. The 'FileUploadWorkflow' is presented as an example, showcasing the use of Temporal's automatic retries and timeouts to ensure task completion even when steps fail. The post also includes a diagram illustrating the high-level workflow process.

Temporal in Space

2/6/2025

This post details a real-world experiment demonstrating Temporal's Durable Execution capabilities by launching a Raspberry Pi running a Temporal worker into space via a weather balloon. The experiment involved writing a Python Workflow that invoked an Activity every five seconds to process sensor data (GPS, altitude). The system successfully handled connectivity loss in the upper atmosphere and resumed operations upon reconnection, showcasing Temporal's fault tolerance. The post also briefly mentions the use of Temporal Workflows for orbital computer simulations in Kerbal Space Program.

Saga Pattern in Microservices: A Mastery Guide

1/31/2025

This post provides a comprehensive guide to the Saga pattern in microservices, explaining its principles, benefits, and implementation strategies. It details how sagas address data consistency challenges in distributed systems, contrasting them with ACID transactions and synchronous/asynchronous communication. The post elaborates on choreography and orchestration approaches, key components like participants and compensation actions, and use cases in e-commerce and finance. It also covers error handling, best practices, common pitfalls, monitoring, and compares sagas with other transaction management patterns. Success stories from Temporal's adoption by companies like ANZ Bank, Maersk, DigitalOcean, and Netflix are highlighted, along with future trends in distributed transaction management.

Simplifying distributed transactions with microservices

1/28/2025

This post elaborates on Temporal's Durable Execution model, specifically focusing on its application to simplifying distributed transactions in microservices. It introduces a concrete e-commerce example with four steps (Fraud Check, Prepare Shipment, Charge Customer, Ship Order) to illustrate the challenges of orchestrating microservices the 'hard way' (ephemeral state, lack of visibility, limited reliability). It contrasts conventional approaches like stateful services and event stores with Temporal's 'durable way'. The post details how Temporal Workflows automatically persist state, provide visibility through the Event History, enable automatic retries, and centralize orchestration logic. It further breaks down key components: Workflows (unique IDs, single execution guarantee), Activities (handling failures, retries, modularity, polyglot support), and the Event History (state persistence, debugging, audit trail). The post concludes by emphasizing Temporal's ability to consolidate events into a single Workflow's Event History for enhanced visibility and simplified operations.

Resilient Event-Driven Architecture for Finance

1/23/2025

This post contrasts traditional choreography-based Event-Driven Architecture (EDA) with Temporal's durable execution model. It highlights the limitations of choreography, such as scattered business logic, hidden coupling, complex error handling, and limited observability. The post then details how Temporal's durable execution centralizes business logic in Workflows, providing stateful management, integrated error handling, and unified observability. It also outlines best practices for implementing EDA with Temporal, including Workflows as Code, modular Activities, error recovery and compensation, dynamic workflow design, and incremental adoption. The post also mentions Temporal Nexus as a feature for inter-service communication.

Reliable data processing: Queues and Workflows

1/22/2025

This post elaborates on Temporal's Durable Execution by contrasting it with traditional persistent queues. It details the challenges of persistent queues, including complex retry logic, dead-letter queue management, message ordering, and duplication. It then positions Temporal Workflows as a solution that abstracts away these complexities, offering a more reliable and less operationally burdensome approach to data processing.

How Temporal Builds Durable Cloud Control Systems

1/16/2025

This post details how Temporal Cloud utilizes its own durable execution model for its control plane operations, specifically for namespace provisioning and rolling upgrades. It introduces the concept of 'entity workflows' tied to specific resources like cells and namespaces to manage their lifecycle and ensure consistency. The post also elaborates on the cell-based architecture for the data plane and the benefits of durable execution for developer productivity and building reliable cloud services.

2024

Simplifying Scalability in Distributed Systems with Workflow Orchestration

12/17/2024

This post elaborates on the benefits of workflow orchestration for scalability in distributed systems. It specifically details how platforms like Temporal simplify the management of state and retries, contrasting this with the complexities of manual implementation in patterns like sagas. The post emphasizes the automatic persistence of workflow state, resumption from failure points, and built-in resilience features. It also includes a case study of Bugcrowd's successful adoption of Temporal for microservices, leading to significant improvements in engagement speed, capacity, and engineering resource savings.

Salesforce streamlines Workflows, cuts code by 95%

12/9/2024

This post details Salesforce's adoption of Temporal, highlighting its use for orchestrating complex distributed systems and reducing manual intervention. It emphasizes the reduction in code (1000 lines to 50) for previously challenging long-running workflows, the benefits of unit testing Temporal code, and the strategic evolution towards building a platform with self-service capabilities. The post also touches on scaling for enterprise needs and the impact on engineering productivity and reliability.

Event-Driven Systems & Loosely Coupled Architectures Explained

12/4/2024

This post elaborates on the benefits of event-driven architectures (EDAs) and how Temporal reduces the complexity and cognitive load associated with them. It specifically details how Temporal's durable execution model handles state, retries, and ensures workflows continue reliably even during failures or event structure changes. It also introduces the concept of version patching for handling evolving event APIs in long-running workflows.

How BPMN and legacy orchestration tools are holding you back

12/4/2024

This post contrasts the limitations of legacy orchestration tools (BPMN, proprietary DSLs, XML/JSON/YAML) with the benefits of modern, code-first workflow solutions. It specifically highlights Temporal's advantages in scalability, reliability (automated retries, state persistence), and developer efficiency (code-first approach, abstraction of error handling and state management). It includes a case study of ANZ Bank's migration to Temporal, detailing an 8x improvement in project delivery speed due to simplified workflows, seamless third-party API integration, and resilience.

How ZoomInfo Built a Scalable Cluster with Temporal Cloud

12/3/2024

This post details how ZoomInfo adopted Temporal Cloud for their dynamic audience management system, handling hundreds of millions of operations monthly. It highlights the challenges of their previous homegrown solution, including throughput limitations and batch processing difficulties. The adoption of Temporal Cloud led to significant reductions in development time (from weeks to days) and debugging efforts (from daily to near-maintenance mode). The migration to Temporal Cloud from a self-hosted cluster resulted in cost savings. The post also showcases the use of Temporal's long-running workflows for their new Copilot product with Account AI, processing large volumes of data and scaling effectively. Key benefits emphasized include workflow-as-code, built-in observability, and the cloud-agnostic nature of Temporal.

Multi-agent Workflow patterns and modern use cases | Temporal

12/3/2024

This post details how Temporal's durable execution capabilities are applied to orchestrate and coordinate multi-agent workflows. It explains how Temporal acts as a conductor for interactions among agents, manages stateful coordination and context sharing, enables reliable asynchronous communication, supports parallel processing and concurrency control, and handles long-running workflows with multi-step processes. It also covers event-driven triggers, signals, and real-time agent coordination, illustrating these concepts with a customer support example. The post emphasizes that Temporal doesn't require building specific code for each agentic framework, acting as a general-purpose backbone for agentic systems.

Events are the wrong abstraction: Rethinking distributed systems

11/12/2024

This post argues that Event-Driven Architecture (EDA) is an outdated and overly complex abstraction for distributed systems, drawing parallels to the geocentric model of the solar system. It details common pitfalls of EDA, including lack of APIs, tight coupling, scattered logic, poor debuggability, ad-hoc state machines, increased latency, race conditions, and system upgradability challenges. The post then presents Durable Execution, as exemplified by Temporal, as a superior abstraction that centralizes business logic, manages state consistently, handles failures gracefully, and allows for flexible process design, ultimately leading to more reliable and maintainable distributed systems.

How to handle fault tolerance in microservices

11/5/2024

This post elaborates on Temporal's Durable Execution by comparing three common approaches to handling failures in microservices: Request-Response (RPC), Persistent Queues, and Workflows. It details the pros and cons of each, emphasizing the limitations of RPC and queues for complex failure management. The post then reiterates Temporal's workflow-as-code approach as a solution for building resilience, citing ANZ Bank as an example.

Modernizing monoliths with Temporal

10/30/2024

This post details how Temporal Workflows can be used to modernize monolithic applications by introducing durable execution for complex, cross-domain processes. It explains the challenges of managing ephemeral state in distributed systems and how Temporal's event sourcing and durable execution model solve this by storing process state durably. The post provides an example of an order management system workflow and highlights the benefits of using Temporal for process visibility, cognitive simplicity, domain division with orchestrated control, management of ephemeral data, testing simplicity, and scalability without data loss. It also discusses how Temporal can be used to add durability and visibility to existing monoliths without breaking them apart.

Spooky Stories: Chilling Temporal anti-patterns (part 2)

10/21/2024

This post details several Temporal anti-patterns and their corresponding best practices, framed within a horror movie theme. It covers: 1. The "MEGA Workflow" anti-pattern, advocating for smaller, single-process workflows and introducing Temporal Nexus as a solution for breaking down complexity. 2. "Arguing With Yourself" by duplicating state machine logic outside of Temporal Workflows, emphasizing that Workflows should manage process state directly. 3. "Hiding Behind the Chainsaws" by prematurely failing Workflows instead of leveraging default infinite retries for Activities, and the importance of understanding when to customize retry policies. 4. "Hiding In a Room With One Exit" by not using compensation patterns for business-level failures, and the need to programmatically handle undoing previous actions. Bonus sections cover "Wandering into a Dark Alley" by not using Temporal's built-in metrics and visibility, and "Vampires Sucking Up All Your Resources" by not using rate limiting to prevent noisy neighbor problems in self-hosted deployments.

Mastering Durable Execution in Distributed Systems

10/14/2024

This post provides a deep dive into the concept of Durable Execution in distributed systems, explaining its benefits over traditional monolithic and microservice architectures. It details how Durable Execution ensures code completion despite failures by persisting state and enabling seamless recovery. The post highlights new programming possibilities such as long-running tasks (sleeping for months) and using instance variables for state management in workflows. It also explains how Durable Execution simplifies implementing patterns like Event-Driven Architecture, Task Queues, and Sagas by offering automatic orchestration and a better developer experience through code readability and UI observability.

Spooky Stories: Money transfers on the edge of town

10/9/2024

This post describes how a financial institution migrated its daily sweep process, which relied on SFTP file uploads and downloads, to Temporal. The migration addressed issues such as frequent upload/download failures, SFTP server overload, and database query failures under heavy load. By using Temporal's event history, the institution can now investigate errors more effectively. The migration was incremental and delivered immediate positive impact, improving stability and resilience.

Durable RAG with Temporal and Chainlit

9/14/2024

This post details how to integrate Temporal with Retrieval-Augmented Generation (RAG) and Chainlit to build Durable RAG systems. It explains how Temporal's durable execution model can be used to orchestrate complex, long-running AI tasks involved in RAG, ensuring reliability and scalability. The post highlights Temporal's role in managing workflow orchestration, scalability, and resource optimization for RAG models, and how Chainlit enhances user interaction with these systems.

What it means to be a Durable Execution Service Provider

2/22/2024

This post introduces the concept of Temporal as a "Durable Execution Service Provider." It elaborates on the core abstraction of Durable Execution, equating it to Temporal's Workflow Execution. The post details the two key components for providing Durable Execution: the SDK and the supervisor service. It explains how the SDK allows developers to build business logic and how the supervisor service (Temporal Server or Temporal Cloud) ensures reliable execution by maintaining state and enabling recovery from failures. The post also highlights Temporal Cloud as a SaaS supervisor option and discusses Namespaces as a service for multi-tenancy.

Managing very long-running Workflows with Temporal

1/25/2024

This post introduces the concept of 'Entity Workflows' for managing very long-running or indefinite processes. It details the 'Continue-As-New' mechanism as a solution to Temporal's Event History limits (50K Events/50MB) for these workflows, explaining its mechanics and considerations such as input design and signal draining. It also briefly mentions the interaction patterns for long-running workflows via Signals and Queries, and the necessity of Workflow Versioning for updates.

Building application reliability on top of infrastructure unreliability

1/12/2024

This post explains how Temporal's Durable Execution model inherently addresses application reliability challenges arising from unreliable infrastructure. It contrasts the complexity of microservice architectures with Temporal's approach, where workflows automatically resume on different processes after interruptions. The post uses an e-commerce example and a simplified two-step process to illustrate the mathematical implications of system reliability and the overhead of traditional microservice patterns. It highlights Temporal's ability to provide end-to-end visibility and guaranteed execution, enabling applications to be reliable even when the underlying infrastructure is not.

2023

Community threads: Is it possible to create an event-based Workflow?

11/27/2023

This post explains how to create event-based Workflows by using `wait_condition` in conjunction with signals. It provides a Python SDK example demonstrating how a Workflow can pause execution until a specific condition is met, which is triggered by an incoming signal. This allows Activities within a Workflow to wait for external events or acknowledgments before proceeding, enhancing the event-driven capabilities of Temporal Workflows.

Nuon gets 36x increase in developer velocity with Temporal long-lived Workflows

11/22/2023

Nuon rebuilt their API using Temporal long-lived workflows, achieving a 36x increase in developer velocity and significant improvements in reliability and insights by managing database objects and interactions within workflows. This post details their experience and best practices for long-lived workflows.

The distributed machine

10/19/2023

This post frames Temporal's Durable Execution as a fundamental shift in computing, akin to the advent of virtual machines and container orchestration. It argues that Temporal creates a 'distributed machine' by abstracting away the complexities of distributed systems, allowing developers to write code as if it were running on a single, highly reliable entity, regardless of the underlying physical or virtual infrastructure. It draws parallels between the evolution of machines (physical -> virtual -> container orchestration) and positions Temporal as the next step, enabling distributed processes to behave as a unified distributed computing system.

Parallelism and Concurrency

8/15/2023

This post clarifies the distinction between concurrency and parallelism, explaining how Temporal leverages these concepts within its durable execution model. It details how Temporal's workflow execution operates on a deterministic event loop for individual workflows, making progress on tasks until they are blocked or completed, and how this model provides fault tolerance and the illusion of parallelism.

What is time-travel debugging?

8/7/2023

This post explains the concept of time-travel debugging, its history, and implementation techniques (record & replay, snapshotting, instrumentation). It then details how time-travel debugging can be applied to production environments, particularly in conjunction with durable execution. The post highlights that durable execution's record-and-replay mechanism, which records external inputs and replays code deterministically, enables time-travel debugging in production by allowing execution history to be captured and analyzed offline. It contrasts this with traditional debugging methods and discusses scenarios where temporary recording in production is feasible. The post also elaborates on the mechanics of durable execution, emphasizing its deterministic nature and how it handles external interactions (volatile functions) and state persistence.

Customer Workflows in Loyalty Programs with Temporal

8/3/2023

This post demonstrates how to implement the Actor Model's core principles (message sending/receiving, state maintenance, actor creation) using Temporal Workflows. It provides a concrete example of a customer loyalty program, detailing the implementation of Signal handlers for 'addPoints' and 'cancelAccount', and the use of `workflow.NewSelector` to manage incoming messages. It also discusses the implications of event history size for long-lived workflows and the use of Continue-As-New.

Saga Orchestration vs Choreography

7/13/2023

This post contrasts the choreography and orchestration patterns for microservice communication within the context of the saga pattern. It explains the trade-offs of each, highlighting choreography's decoupling and potential for unclear control flow, and orchestration's centralized control and debugging benefits but also its single point of failure. The post then positions Temporal's durable execution as an orchestration solution that avoids the single point of failure by logging program progress, allowing for seamless continuation on other machines. It emphasizes that Temporal's workflow logic is expressed in code, simplifying development and ensuring failure resilience.

25 key terms for speaking Distributed Systems and Temporal

6/29/2023

This post defines and explains 25 key terms related to distributed systems and Temporal, including core concepts like concurrency, scalability, reliability, eventual consistency, strong consistency, CAP Theorem, ACID, atomicity, isolation, durability, durable execution, idempotent function, deterministic function, platform, application, and event sourcing. It also defines Temporal-specific terms such as Temporal, Worker, Workflow, Activity, Query, Signal, retry, Cluster, and History. The post aims to provide a foundational understanding for engaging in conversations about distributed systems and Temporal.

Workflows as Actors: Is it really possible?

6/28/2023

This post details how Temporal Workflows can be used to implement the Actor Model. It explains the core tenets of the Actor Model (message passing, creating other actors, maintaining state) and maps them to Temporal's features: Signals and Updates for message passing, Child Workflows for creating other actors, and the inherent statefulness of Workflows for maintaining state. It also discusses the use of Queries for retrieving state and the implications of long-running Workflows.

Saga Design Pattern Explained for Distributed Systems

5/24/2023

This post explains the saga design pattern and how Temporal's Durable Execution inherently supports its implementation. It details how Temporal automatically manages state, retries, and recovery, simplifying the developer's task to defining compensation logic for activities. The post contrasts sagas with two-phase commit and provides pseudocode and Java/Go examples of implementing compensation logic within Temporal Workflows. It also briefly mentions the importance of idempotency for safe retries.

Getting started with small-step operational semantics

4/3/2023

Introduces SagaLang, a minimal language for specifying sagas with forward and backward recovery, and uses Small-Step Operational Semantics (SOS) to formally define their execution. Details three iterations of SagaLang: basic execution, forward recovery with retries, and backward recovery with compensation. Highlights the developer's responsibility for ensuring idempotency and correct compensation logic for achieving transaction-like guarantees.

How Durable Execution works

3/16/2023

This post details the client-server-worker architecture of Temporal, explaining how the client initiates workflows, the server maintains state via an event history database, and workers poll for and execute tasks. It elaborates on the sequence of events during workflow execution, including `WorkflowExecutionStarted`, `WorkflowTaskScheduled`, `WorkflowTaskStarted`, `ActivityTaskScheduled`, and `ActivityTaskCompleted`. The post also touches upon the role of the Temporal CLI for development and the deployment of components to production environments like Vercel and Render.

Building Distributed Systems in Node

1/20/2023

This post introduces the concept of durable execution and demonstrates its practical application using Temporal's TypeScript/JavaScript SDK. It explains how durable execution simplifies distributed systems by persisting execution state, handling failures automatically, and enabling long-running operations. The post provides a concrete example of a food delivery app built with durable JavaScript, showcasing how to start workflows, handle retries for external service calls, and use signals and queries for interaction with long-running workflows.

The linear developer experience

1/10/2023

This post introduces the concept of a 'linear developer experience' for complex distributed systems, advocating for frameworks that treat workflow orchestration as 'distributed systems as a service'. It argues that such systems should handle state persistence, retries, error recovery, and timing automatically, allowing code to remain a readable sequence of steps. This approach aims to avoid the 'tipping point' where developer experience deteriorates as systems scale, contrasting it with traditional event-driven architectures and distributed choreography that can lead to infrastructure bloat and debugging challenges.

2022

Do... or do not: Why Yoda doesn't use microservices

11/9/2022

This post elaborates on the challenges of state management in microservices, contrasting them with the transactional guarantees of ACID databases. It introduces the concept of a dedicated state management platform (like Temporal/Cadence) as a solution, highlighting its ability to provide durable execution and abstract away the complexities of distributed state management. The post emphasizes how this approach allows developers to benefit from microservices' agility without sacrificing reliability, drawing parallels to atomic database transactions and the 'do or do not' principle.

Paper summary: End-to-end arguments in system design

8/16/2022

This post applies the End-to-End Argument from Saltzer, Reed, and Clark to system design within the context of Temporal. It uses the example of reliable file transfer to illustrate how critical functions like failure detection and mitigation should be primarily implemented at the application layer for completeness and correctness, while the platform layer can provide optimizations. This principle guides the placement of functionality in Temporal systems to ensure robustness and reliability.

Caching API Calls with Long-Lived Workflows in Temporal

5/26/2022

This post demonstrates how to use Temporal's long-lived Workflows and Activities to implement a stateful caching layer for API calls. It details the creation of a Workflow that periodically fetches exchange rates from an external API using an Activity, stores the rates in memory, and exposes them via a Query. The post also explains the importance of `Continue-As-New` to manage the Workflow's event history limit and how to pass state between Workflow executions. It highlights Temporal's ability to handle state persistence, scheduling, and querying without explicit database or cron job management.

Introduction to Temporal Workflows

1/24/2022

This post introduces Temporal Workflows as the core abstraction for reliability and scalability, contrasting them with regular function executions and demonstrating how they simplify complex, long-running processes like sending reminder emails. It highlights the developer experience of writing code that looks like a specification and reliably executes to completion, akin to database transactions.

2021

Inversion of Execution

12/1/2021

This post introduces and elaborates on the "Inversion of Execution" model as the core architectural principle behind Temporal. It contrasts this with traditional execution models where a system directly executes application code. The post details how Temporal Server orchestrates workflow and activity tasks, which are then executed by client-side Workers. It highlights the role of Temporal SDKs in managing this interaction and reconstructing client state after failures. The benefits of this model, including reliability, security, and scalability, are discussed in detail, using a bank transfer example to illustrate the mechanics.

Tips on defining Workflows

10/11/2021

This post provides a general introduction to workflows and their importance, focusing on different approaches to defining them. It details the pros and cons of Domain-Specific Languages (DSLs) such as flowcharts, forms, and markup, and contrasts them with general-purpose programming languages. The post uses a personal morning routine analogy to illustrate workflow concepts like iteration, domain-specificity, failure handling, and recovery. It emphasizes that while DSLs are good for small to medium complexity, they can become unwieldy for complex definitions, often requiring embedded code. The post sets the stage for a subsequent article that will delve deeper into developing real-life workflow solutions.

Maxim and Samar on Gremlin's Build Things Podcast

10/7/2021

This post elaborates on Temporal's Durable Execution model by contrasting orchestration with choreography, highlighting how Temporal enables developers to write fault-tolerant code that is guaranteed to finish execution. It details how Temporal handles long-running tasks, sleep calls, and process failures by persisting and replaying workflow state. The post also introduces the concept of 'fault-oblivious programming' where applications are not even aware that a fault has occurred. It further explains the role of Task Queues with infinite retries, exponential backoff, and heartbeating for long-running tasks, as well as the ability to maintain local state within workflows. The discussion includes a customer example of a loyalty program implementation and emphasizes the benefits of redesigning applications end-to-end for Temporal adoption to achieve significant improvements in simplification and code reduction. The importance of durability and statefulness for applications is also stressed, suggesting Temporal as a solution for managing complexity beyond traditional databases and queues.

What does long running mean?

8/13/2021

This post reframes the understanding of 'long running' work in the context of Temporal. It contrasts traditional definitions based on time cutoffs with Temporal's approach, which handles both short, transactional orchestrations and infinitely long processes. It explains how Temporal's long polling mechanism enables fast responses for shorter tasks and how event history persistence and the ContinueAsNew API allow for infinitely long workflows. The core contribution is positioning 'long running' as a proxy for 'handling uncertainty' in distributed systems, covering network boundaries, horizontal scaling, and migration coordination.

Maxim Fateev on the OSS Startups Podcast

8/6/2021

This post provides a historical overview of Temporal's origins, tracing its lineage from AWS Simple Workflow Service and Azure Durable Functions. It details the transition from Cadence to Temporal, emphasizing the decision to open-source the project for long-term sustainability and industry-wide adoption. The post also discusses the challenges of building a startup, particularly in hiring specialized distributed systems engineers, and highlights the importance of customer pain points and the founders' deep experience in establishing trust and product-market fit. The core concept of Temporal as a stateful computation engine that preserves and guarantees the state of a process is reiterated.

Media processing Workflow solution

7/19/2021

This post details a media processing workflow solution, showcasing how to build a robust system using Temporal for complex, stateful, and fault-tolerant orchestration of tasks like requesting, downloading, transforming, and processing sensor data. It demonstrates the use of Temporal Activities for individual steps and Workflows for orchestrating these steps, including handling various failure scenarios like connectivity issues, API downtime, and worker crashes. The post also highlights the use of `workflow.ActivityOptions` for configuring timeouts and retry policies, and `workflow.CreateSession` for scheduling related activities on the same worker instance.

Workflow Engine Design Principles with Temporal

4/5/2021

This post provides a foundational explanation of workflow engine design principles, drawing parallels to Temporal's architecture. It details the historical evolution leading to Temporal, defines the core properties of a workflow (resilient, executes tasks, reacts to events, deals with time), and elaborates on key components like task queues and timer queues. A significant focus is placed on the necessity of transactional updates across state, task queues, and timers to ensure consistency and prevent race conditions. The post also outlines Temporal's scalability approach, emphasizing scaling out the number of workflows and the use of sharding for distributed persistence and hosts.