TechPrufer
Get started
Temporal
Temporal logoTemporal · Feature trail

Durable Execution

120 posts · Sep 2020 – Sep 2026

What this trail covers

Temporal Workflows provide a durable orchestration layer for complex business processes and event-driven systems. This post defines Durable Execution as 'crash-proof execution', emphasizing its benefits in improving reliability, simplifying code by focusing on goals over potential problems, and accelerating development by eliminating complex error-handling logic. It details the four key characteristics: virtualization of execution across processes/machines, no time limitation for execution, automatic preservation of application state, and hardware agnosticism. The post highlights how Durable Execution abstracts away failures, allowing developers to focus on application logic rather than anticipating and handling every possible failure mode.

All Temporal trails
  1. Authentication is a long-running WorkflowThis post details how KERNEL models long-running authentication connections as durable Workflows, incorporating human intervention and external approvals as explicit states within the Workflow lifecycle. It highlights the use of child Workflows for individual login attempts and the separation of concerns between Go for durability and TypeScript for agent execution. The post also discusses strategies for maintaining authenticated sessions, including recurring health checks, automatic reauthentication, and circuit breakers for repeated failures.
  2. Stop restarting your testsThis post details how Temporal's Durable Execution can be applied to orchestrate end-to-end test suites, specifically using Playwright as an example. It introduces the concept of treating each independent test execution (e.g., a specific spec file against a specific browser) as a Temporal Activity. This allows for individual retries of failed test executions without re-running the entire suite, preserving progress and reducing feedback loops in CI/CD pipelines. The implementation involves a wrapper around the existing test runner, where a Temporal Workflow fans out Activities to execute individual tests. Key technical aspects include the use of `Promise.allSettled` to ensure all executions reach a terminal state, the independent nature of Activities with their own retry policies, and the separation of test discovery from Workflow logic. The post emphasizes that Temporal does not fix flaky tests but provides a robust mechanism for handling their transient failures.
  3. The thread is the Workflow: Durable AI agents without changing Agent codeThis post details how Manetu's AgentVisor runtime leverages Temporal Workflows to provide durable execution for AI agents without requiring changes to the agent code itself. It maps Agent Protocol concepts (Thread, Run, Checkpoint, Interrupt) to Temporal implementations (Workflow instance, Workflow Update + Activity, Workflow state, Pending Workflow state + signal). Key engineering challenges addressed include managing state for long-lived agent threads within Temporal's history limits (retaining only the latest checkpoint and relying on replay), differentiating between crash-recovery and human-in-the-loop resumes, handling Continue-As-New rollovers synchronously, and resolving worker identity issues by providing explicit build identifiers. The post also highlights the integration of Temporal's Payload Codec for transparent encryption of agent state.
  4. How LlamaIndex uses Temporal to scale reliable document orchestrationThis post details LlamaIndex's migration from RabbitMQ to Temporal for orchestrating complex document parsing pipelines. It specifically showcases how Temporal Workflows are used to implement a robust concurrency control system (semaphores) to manage shared resources like GPU pools and project-specific job limits. The implementation leverages Temporal's atomic start-workflow operations and durable state to ensure fairness, backpressure, and reliable resource allocation, overcoming limitations of their previous system which required significant custom logic for status tracking, heartbeats, and fairness primitives. The post provides code examples of Temporal Workflows used as locks and how they are invoked to acquire and release permits, demonstrating a practical application of Temporal's durable execution for managing distributed system concurrency.
  5. Better together: The durability layer for your AWS agentic stackThis post details how Temporal adds a durability layer on top of AWS agentic building blocks (Bedrock, Strands Agents SDK, AWS Lambda, AgentCore Runtime). It explains how Temporal integrates with the Strands agent loop to provide automatic state management for model invocations and tool calls, making each step recorded in Workflow history and survivable across failures. It also highlights the use of Temporal's Serverless Workers feature with AWS Lambda for scalable, cost-effective compute, and discusses the conceptual similarity and integration roadmap with AgentCore Runtime. A practical example of a Loan Underwriting agent demo is provided to illustrate these concepts.
  7. The “cool new stuff” trapThis post details the development of an AI agent to convert STL files to editable CAD, highlighting the 'trap' of relying solely on AI for complex problem-solving. It emphasizes the critical role of a robust validation layer (measuring face counts, analytic surfaces, editability scores, etc.) in driving the conversion process. The architecture evolved from an agentic loop to a more direct pipeline: evaluate passes, build conversion plan, build candidate, select best valid candidate. The post also touches on using external tools like Fusion as candidate backends, underscoring that measurable outcomes are paramount.
  8. Durable RAG and agents: MongoDB and Temporal doing it better togetherThis post details how Temporal's Durable Execution is applied to a RAG knowledge base and research agent architecture. It highlights the use of Temporal Workflows for reliable data ingestion, including chunking, embedding, and indexing into MongoDB Atlas. It also explains how Temporal orchestrates the agent's reasoning loop, ensuring state is preserved across fallible external calls to models, tools, and databases. The post emphasizes the benefits of Durable Execution in eliminating the need for systems like Kafka for buffering and enabling seamless infrastructure maintenance for long-running jobs.
  9. Chaining Activities — from text to vectorsThis post details a specific implementation of a data processing pipeline using Temporal Workflows and Activities. It showcases chaining multiple Activities for tasks like text embedding, vector caching (using Redis), vector storage (in Pinecone), and cleanup. It highlights the use of `case_id` for idempotency in Postgres and upsert semantics in Pinecone, along with an `embedded` flag for reconciliation. The post also demonstrates how to manage external dependencies like embedding models and vector databases through provider interfaces and discusses the importance of cache cleanup.
  10. When the human is the Workflow: Durable Execution in harsh physical field operationsThis post details the application of Temporal's Durable Execution to harsh physical field operations, specifically for construction and facilities management. The core contribution is modeling each shift as a single Temporal Workflow, addressed by a deterministic ID derived from the schedule. This design decision simplifies the mobile app's interaction by issuing Updates against the Workflow, leveraging Temporal's single-instance semantics. Key technical patterns highlighted include: 1. Replacing GPS with BLE beacons for reliable location tracking in unreliable environments. 2. Implementing idempotency using `workflowIdConflictPolicy: WorkflowIdConflictPolicy.USE_EXISTING` and `workflowIdReusePolicy: WorkflowIdReusePolicy.ALLOW_DUPLICATE_FAILED_ONLY` for duplicate requests. 3. Utilizing the `Signal-With-Start` pattern to handle offline-first operations and ensure events are processed even if the initial 'start shift' signal is lost. 4. Employing an in-workflow request queue with sequence numbers and `latestWins` logic for ordering and deduplication of Updates, serving as an audit trail. 5. Implementing compliance-as-code with Update validators to enforce business rules before state mutation. The post also contrasts this approach with conventional methods, emphasizing the deletion of 'sediment' like cron jobs and reconciliation logic in favor of a durable workflow as the source of truth.
  11. Durable, flexible multi-agent systemsThis post details the use of Temporal's Durable Execution to build a multi-agent system for fleet delivery, demonstrating its ability to orchestrate agents across different frameworks (ADK and LangGraph) and handle human-in-the-loop interactions. It highlights how Temporal's event history and durable state management allow agents to wait for human input for extended periods without consuming compute resources, and how patterns like separate task queues and Continue-as-New address challenges like resource starvation and event history growth in long-running loops.
  12. How Temporal powers workflows at HeyGenThis post details how HeyGen migrated from a legacy internal job system to Temporal for orchestrating their video generation workflows. It explains their modeling of video generation as Temporal Workflows and jobs as Activities, highlighting the benefits of Temporal's state persistence and automatic retries over their previous database-centric approach. The post describes how they handle concurrent scene processing within a single workflow using semaphores and child workflows for operations needing independent failure domains. It also elaborates on their multi-layered fairness approach, combining a custom scheduling layer with Temporal's task priority. Furthermore, it outlines the shared workflow platform they built, including reusable workflow modules and a common interceptor stack for identity propagation, error classification, metrics, and search attributes.
  13. I spent two months screen-sharing to ship one integration. So I rebuilt it on Temporal.This post details the construction of an egress-only proxy application that leverages Temporal as a durable orchestration layer for cloud-to-edge messaging. It replaces traditional queuing and polling with a single outbound connection to the Temporal Service, enabling remote configuration updates and eliminating inbound firewall requirements. The solution addresses the challenges of unreliable connections, dropped messages, and difficult debugging in firewalled environments by providing out-of-the-box durability, reliability, and visibility.
  14. The bot that lied to cricket fans with a straight face: Temporal x San Francisco UnicornsThis post details the implementation of a fan engagement bot using Temporal Workflows to manage stateful operations like reminders, live-score polling, and post-match summaries. It highlights the use of four distinct Workflows (`conversationTurnWorkflow`, `matchReminderWorkflow`, `matchScoreSubscriptionWorkflow`, `postMatchSummaryWorkflow`) to handle different aspects of the bot's functionality. The post emphasizes how Temporal's durable execution ensures these operations continue seamlessly even if Workers fail, a critical requirement for a system operating across time zones and during live events. It also discusses challenges and solutions related to WhatsApp's messaging platform constraints, such as the 24-hour messaging window and webhook verification. A specific failure mode is described where a hardcoded free-tier limit caused the bot to stop fetching live scores, leading to stale data, and how this was resolved by correcting the limit and ensuring the recovery mechanism itself didn't become a blocker.
  15. The foundation: Why Temporal for a data pipeline?This post details the application of Temporal's Durable Execution for building a PII-compliant data pipeline. It outlines the design decisions and justification for using Temporal to meet stringent enterprise requirements for resilience, consistency, availability, and observability. Key contributions include: demonstrating how Temporal's out-of-the-box retry and resume logic simplifies resilience compared to naive approaches; explaining how Event History ensures consistency by recording Activity results and requiring Activity idempotency; illustrating how Workflow replay provides availability by resuming from the last known state after worker restarts or instance failures; and highlighting the Temporal Web UI's role in providing auditability for pipeline steps. The post also provides a high-level architecture overview and a mapping of pipeline requirements to a Temporal Workflow, including the use of Standalone Activities for event ingestion and discrete Activities for PII checking, classification, storage, embedding, and vector database interaction.
  16. Orchestration powered by Temporal: Wrap, don’t rewriteThis post details Zscaler's approach to building a next-generation automation platform by using Temporal Workflows as a durable orchestration layer above heterogeneous execution engines (Ansible, Terraform, Helm, ArgoCD, Crossplane, Kubernetes Cluster API, AWS CloudFormation, Jenkins). The core contribution is the 'brownfield wrapping' pattern, where existing tools are wrapped as leaf-level Activities in Temporal Workflows without refactoring them. This provides durability, enables cross-engine workflows, allows for incremental adoption, and preserves direct usability of the underlying engines. The post emphasizes that Temporal Workflows act as a contract that survives worker failures, making complex sequences recoverable and retries configurable. It also positions this pattern as a foundation for integrating AI workflows as new leaf activities.
  17. I had two weeks, an 8-foot keyboard, and a Wordle WorkflowThis post details the implementation of a Wordle game where the Temporal Workflow serves as the sole source of truth for game state. It describes the architecture where the Workflow handles game logic, guess validation, and state updates via Workflow Updates. The post also covers the practical challenges and solutions for integrating with physical hardware (an 8-foot keyboard) and displaying Temporal's event history on a holographic fan in a live conference booth setting. It highlights the iterative development process, including live UI mutations and operational changes like a safer deploy process, all while maintaining the core durability of the Temporal Workflow.
  18. LangGraph in production: Temporal's LangGraph Plugin adds Durable ExecutionThis post introduces a LangGraph integration for Temporal Python SDK (v1.27.0+) and LangSmith integrations for Python and TypeScript. The LangGraph integration allows LangGraph agents to run as Temporal Workflows, with LangGraph nodes executing as Temporal Activities. This provides durable execution, automatic recovery from failures, and cost-effective, durable human-in-the-loop steps using Temporal's signals and timers. The LangSmith integration propagates trace context across Temporal boundaries for end-to-end tracing of agent runs.
  19. Why a context engine needs Durable Execution: How Unblocked rebuilt its ingestion backbone on TemporalThis post details Unblocked's migration of their AI context engine's ingestion backbone from a DIY actor model to Temporal's Durable Execution. Key technical contributions include: collapsing pages of orchestration code to lines of Temporal Workflow code using a thin Kotlin shim over the Java SDK, minimal abstraction philosophy exposing Temporal primitives, opinionated retry and exception handling tuned to ingestion patterns, and the adoption of Temporal for various data sources (Asana, CI, Slack). The post highlights the benefits of deployment-free long-running jobs, proper scheduling of background jobs, unified handling of user-triggered and cron flows, and built-in observability via the Temporal UI. It also provides production deployment details: self-hosted on Kubernetes with 12 History and 12 Matching replicas, Aurora Serverless persistence, and KEDA autoscaling per queue. The migration is ongoing, with code ingestion transitioning from Prefect.
  20. Why I needed Durable Execution to read a toy manualThis post details the application of Temporal's Durable Execution for a personal project translating toy manuals. It showcases: 1. Fan-out pattern for parallel OCR processing per page. 2. Use of Retry Policies for unreliable translation APIs. 3. Heartbeating for long-running site generation Activities. 4. Structuring LLM interaction within Activities using Pydantic for validation and keeping nondeterministic code out of the Workflow definition. It highlights how Temporal's event history and state management enable pipeline continuation after failures without manual intervention.
  21. Why teams run Temporal workers with EncoreThis post details how Encore integrates with Temporal to provide automated infrastructure provisioning and orchestration for backend services. It highlights how Encore services can host Temporal workers, simplifying deployment and management. The post also discusses how AI agents, which are long-running processes, can benefit from the combined durability of Temporal and the infrastructure management of Encore. It provides code examples demonstrating the integration of Encore's typed APIs and configuration with Temporal workflows and activities.
  22. How Rapidflare built a million+ document ingestion pipeline for Agents on TemporalThis post details Rapidflare's implementation of a million+ document ingestion pipeline using Temporal. Key technical contributions include: 1. Using Temporal Activity Heartbeats for long-running crawlers, sending meaningful progress updates instead of generic heartbeats. 2. Setting `maximum_attempts=1` on crawler Activities, relying on internal crawler retry logic to avoid discarding progress. 3. Utilizing Cloud Storage as a data bus for large artifacts (beyond Temporal payload limits) to enable state sharing between Activities. 4. Implementing a sliding window mechanism with Child Workflows and `workflow.wait_condition` to control concurrency and prevent overwhelming downstream rate-limited services. 5. Employing Continue-as-New at page boundaries to manage large Workflow state and keep history limits in check. 6. Using deterministic Child Workflow IDs for predictable conflict resolution. 7. Designing a source-agnostic pipeline with three Temporal Workers on separate Task Queues for isolation (Ingestion, Enrichment, Status sync).
  23. How Sherlocks AI uses Temporal to orchestrate AI agents for incident resolutionThis post details how Sherlocks AI leverages Temporal's Durable Execution for orchestrating AI agents in incident resolution. It describes four distinct workflow patterns: Knowledge Graph construction (replacing cron jobs with scheduled Workflows), Infrastructure scanning (handling long-running tasks with retries and checkpointing), AI agent investigations (managing complex multi-stage processes with specialized agents), and Event ingestion (ensuring durable acknowledgment of triggers). The post highlights the benefits of Temporal's retry policies, scheduling, human-in-the-loop capabilities via Signals, and observability for these patterns. It also emphasizes Temporal's model-agnostic approach, allowing flexible use of different LLM providers and models within Activities. Structural choices include per-tenant Workflows for isolation, scaling Workers as separate processes, and using a Codec Server for data encryption.
  24. How we built it: Running Replay on TemporalThis post details the architecture and implementation of a conference registration system for Temporal's Replay conference, built on Temporal. It highlights the use of Workflows for managing the entire lifecycle of an order and individual tickets, treating them as processes rather than database rows. The post emphasizes the use of Activities for interacting with external systems (payment processors, email APIs, databases) with configurable retry policies and idempotency keys. It also discusses the pattern of spawning a separate Ticket Workflow for each ticket, managed by an Order Workflow, and the use of Signals and Queries for external communication. The system demonstrates how Temporal handles long-running, stateful processes with complex event-driven logic, including waiting for events over months.
  25. Stop failing on the path to production: A better way for agentic platformsThis post positions Temporal's Durable Execution as a foundational harness for building agentic platforms. It outlines the requirements for successful agent architectures (memory, resilience, programming model, auditability, visibility, unlimited runtime, data typing, LLM/tool orchestration, customization, full orchestration control) and maps them to Temporal's core primitives (Workflows, Activities, Signals, Queries, Updates). It emphasizes the importance of secured data and deployed agents over desktop agents, and provides a reference sample harness and links to related resources for building agentic platforms with Temporal.
  26. How Kelet AI built a durable agent that debugs durable agents.This post details the architecture and implementation of Kelet AI's durable agent for diagnosing AI quality failures, built on Temporal. It introduces a four-level Workflow hierarchy (Session, Signal, Agent Aggregation, Investigate Issue) to address the challenge of identifying root causes across multiple AI agent sessions. Key Temporal primitives utilized include long-running state, durable execution, event-driven coordination via Signals, and cross-workflow gating. The post also describes a Temporal plugin for integrating Kelet's monitoring, which propagates session attributes via Temporal headers for deterministic replay and survival of worker restarts and `continue_as_new`.
  27. How Monk migrated 100+ workflows from Inngest to TemporalThis post details Monk's incremental migration of over 100 Inngest workflows to Temporal. It outlines a four-step process: characterization with tests, scaffolding Temporal components, cutover via feature flags, and removal of the old system. Key technical insights include mapping Inngest's cron triggers, concurrency, and retries to Temporal's Schedules, Activity configurations (startToCloseTimeout, retry), and Workflow IDs. It highlights the critical difference between Inngest's step-based invocation and Temporal's event history replay, necessitating the refactoring of non-deterministic code (e.g., Date.now(), random picks) into Activities. The post provides a concrete example of migrating a scheduled drift check workflow, demonstrating how Inngest's `createFunction` with inline logic is transformed into a Temporal Workflow orchestrating thin Activities, with the Schedule object managing the cron trigger. It also details the correct mapping of Inngest retries to Temporal's `maximumAttempts` and the importance of using Workflow IDs for idempotency.
  28. Coordinate Access to Shared Resources with a Distributed Lock Built on Temporal WorkflowsIntroduces a validated pattern for implementing distributed locks using Temporal Workflows. Each lock is represented by a short-lived child workflow, leveraging Temporal's atomic workflow ID uniqueness for lock acquisition. Orphaned locks are recovered via lease timeouts. This pattern eliminates the need for external databases, central rate limiters, or shared state outside of Temporal.
  29. Diving into the AI iceberg: What lies beneath your AI agentsThis post details how Temporal orchestrates a multi-agent AI research workflow, highlighting its role in coordination, human-in-the-loop interactions, parallel execution of search activities, automatic retries for transient failures, and ensuring durable execution for long-running processes. It provides a simplified code example of the workflow structure and emphasizes the value of Temporal in making such complex AI applications resilient and observable.
  30. How Coverwatch uses Temporal to orchestrate AI-powered insurance workflowsThis post details how Coverwatch uses Temporal to orchestrate AI-powered insurance workflows, combining AI agents, human review, carrier communication, and operational follow-up. It highlights the use of Temporal's durable execution guarantees with agentic automation through Pydantic-AI integration, and how Temporal Signals are used to integrate human review into the durable flow. The post also emphasizes the value of workflow history for debugging and iteration by both engineers and AI coding agents.
  31. How to build deep research agents using Temporal and BraintrustThis post details how to build resilient multi-agent systems, specifically deep research agents, by mapping agents to Temporal Activities and leveraging Braintrust for observability and evals. It highlights parallel execution of searches with graceful degradation, structured data contracts, and configurable timeouts to handle LLM API variability and partial search failures.
  32. How to build a resilient platform control plane with TemporalThis post details building a platform control plane using Temporal's Entity Workflow pattern. It introduces a one-to-one mapping of Workflows to managed resources, covering state management with `ResourceState`, message handling via Updates, Signals, and Queries, and the main loop for serializing side effects. Key implementation details include using `workflow.wait_condition` for periodic ticks, `_process_next_command` for serializing mutations, and `workflow.continue_as_new` for history rotation. It also highlights the use of Update validators and application-level `request_id` for idempotency.
  33. How groundcover built a durable alert dispatch system with TemporalThis post details groundcover's implementation of a durable alert dispatch system using Temporal. Key technical contributions include: a two-level Workflow architecture (orchestrator and per-alert Workflows), deterministic Workflow IDs for `SignalWithStartWorkflow`, signal batching within per-alert Workflows using `ReceiveAsync` and a small sleep window, a fan-out mechanism with semaphore-controlled goroutines and per-alert ordering preservation, backpressure management via `CountWorkflow` and dynamic scan limits, and a two-tier state management approach (in-memory for performance, PostgreSQL for persistence). The system also uses `Continue-as-New` for history bounding and Temporal Retry Policies for Activity failures.
  34. Human-in-the-Loop Approval WorkflowsThis post details how to build a human-in-the-loop approval system using Temporal's Durable Execution. It highlights how Temporal Workflows can durably wait for decisions, automatically enforce SLAs, trigger escalations without schedulers, and maintain complete audit trails, addressing common failure modes of traditional approval processes such as unanswered requests, missed deadlines, lost context across restarts, and incomplete audit trails. It contrasts this with the complexity of maintaining multiple systems (database, cron job, notification service, custom reconciliation logic) for similar functionality.
  35. What does it take to build a customer support experience your users won't hate? Ask Bitovi.This post details how Bitovi built a stateful AI agent for customer support using Temporal Workflows and Activities. The ReAct agent loop is implemented as a Temporal Workflow, with LLM inference and memory extraction handled by Activities. The post emphasizes the benefits of Temporal's Durable Execution for handling long-running, stateful conversations and integrating external memory systems like AWS Bedrock AgentCore Memory. It highlights the isolation of nondeterministic LLM inference within Activities and the use of structured output with validation and retries for reliability. The post also provides a code snippet for a `ThoughtActivity` and links to the full implementation on GitHub.
  36. From FileUploadWorkflow to creative OS: How Layer scaled on TemporalLayer.ai details their evolution from using Temporal for simple file uploads to building a sophisticated visual pipeline builder on top of it. They use Temporal child workflows for each node in their DAG-based pipeline, with a parent workflow orchestrating execution and using `workflow.wait_condition` against a shared durable data bus (`all_values` dict) to manage inter-node dependencies and data flow. They also highlight the adaptability of their `RunInferenceWorkflow` to handle multiple modalities (image, video, 3D, audio) without redesign, incorporating modality-aware timeouts and parallel post-creation steps. The integration of an MCP connector allowing LLM agents to directly invoke Layer workflows is also discussed as a key advancement for autonomous operations.
  37. From agent zoo to agent orchestra: The benefits of Temporal as your enterprise agentic control planeThis post details how Temporal can be used as an enterprise agentic control plane, extending the concept of Durable Execution to AI agents. It categorizes agents into Conversational, Event Stream Processors, Ambient, and Agent-to-Agent interactions, and explains how Temporal Workflows provide essential features like guardrails, human-machine interaction, versionability, durability, cost management, visibility, and auditability for these agent types. It introduces Temporal Nexus as a mechanism for coordinating agents across namespaces and discusses Workflow Versioning for managing changes in AI agent logic. The post also touches upon operational realities like LLM cost management and retry policies.
  38. How XY builds an AI agent orchestration platform for healthcare with TemporalThis post introduces XY's AI agent orchestration platform for healthcare, built on Temporal. Key contributions include: a YAML-based DSL interpreted by a single generic `DSLWorkflow` class, enabling declarative workflow definition; an AI Planner Agent that translates natural language into executable YAML and generates activities; a dynamic Activity Factory for runtime assembly of customer-specific integrations; and a deferred continue-as-new mechanism for handling large-scale batch processing. The post also highlights the benefits of Temporal's vocabulary for team alignment and its role in managing data sensitivity and human-in-the-loop processes.
  39. What agentic AI borrowed from microservices (and made worse)This post draws parallels between microservices architectures and agentic AI, highlighting how concepts like decomposition, orchestration, event-driven architectures, and state management are reappearing in the AI space. It specifically discusses how Temporal's durable execution model addresses the challenges of long-running AI workflows, short-term memory, and the need for event-driven architectures, positioning Temporal as a solution for building reliable and scalable AI agent systems.
  40. Introducing Temporal and agentic sandboxes: The OpenAI agents SDKThis post details the integration of Temporal's Durable Execution with OpenAI's Agents SDK to provide fault-tolerant execution for sandbox agents. It introduces an `AgentWorkflow` that wraps the `SandboxAgent` to persist state and idle efficiently, a `SessionManagerWorkflow` for orchestrating agent sessions, and a TUI for client interaction. Key technical contributions include the use of `workflow.wait_condition` for durable idling, forking workflows across different sandbox backends by snapshotting state, and wrapping sandbox operations as Temporal activities via `temporal_sandbox_client`.
  41. AI reliability is a decade-old problem. And we’re still only solving half of itThis post argues that AI reliability is a decade-old problem, with the industry focusing on model accuracy while neglecting system resilience for AI agents. It highlights the shift from AI systems that suggest to those that act, leading to critical failures when autonomous, multi-step workflows encounter issues. The author emphasizes the need for infrastructure that provides a 'digital bookmark' for workflows, enabling checkpointing, recovery, and resumption. This is framed as the core principle behind Temporal's Durable Execution, which is presented as the missing foundation for production AI agents to handle failures gracefully, preventing catastrophic outcomes like data loss or unauthorized actions. The post contrasts the historical focus on model reliability (accuracy, bias, hallucination) with the current need for system reliability in AI agents executing long-running operations.
  42. Why your platform control plane belongs on TemporalThis post introduces the application of Temporal's Durable Execution to platform control planes. It details how Workflows can manage the entire infrastructure lifecycle (creation, monitoring, upgrading, modification, decommissioning) by providing a durable, multi-step, human-in-the-loop execution layer. It highlights how Temporal's ability to pause and replay Event History on worker recovery eliminates inconsistent or orphaned states. The post also explains how Temporal makes scheduled automation robust with out-of-the-box retries, timeouts, and observability, and enables true self-service by supporting long-running execution and human-in-the-loop patterns via Temporal Queries and Signals.
  43. How I shipped a Temporal-powered product in three weeks, one year out of collegeThis post details the implementation of a property management system using Temporal Workflows. It showcases a polling workflow that initiates child workflows for maintenance requests, managing state, timeouts, and human interactions via Signals. The post highlights the use of Activities with retry policies for interacting with unreliable external systems and the benefits of child workflows for independent lifecycle management. It also discusses an anti-pattern in polling loops and presents a more idiomatic approach using server-side Activity retries to manage workflow history size and improve efficiency.
  44. Building AI agents that overcome the complexity cliffThis post introduces a taxonomy of AI agent capability levels (Reflexive, Conversational, Orchestrated, Coordinated, Autonomous) and identifies 'the complexity cliff' where most agent frameworks fail due to compounding challenges of inevitable failures and slow iteration speed in long-running, multi-step workflows. It highlights that Durable Execution is required above this cliff to address failure recovery (resuming from exact failure points with full context) and efficient iteration (testing prompt variants without re-running entire workflows). Temporal's Durable Execution is presented as the solution, offering automatic state persistence, replay capability, and history branching for fast, efficient parallel experiments, thereby enabling agents to move beyond the complexity cliff to higher levels of capability.
  45. Trusting AI agents: A reinsurance case studyThis post details the implementation of a multi-agent AI system for reinsurance data processing using Temporal Workflows and Activities. It highlights the use of Temporal Signals for human-in-the-loop confirmation of tool executions and agent completions, and Temporal Workflows as a central router (Bridge Workflow) and inter-agent data store. The post also discusses principles for building AI agents, including designing for human-in-the-loop, using modular sub-agents, and keeping tool arguments simple, all within the context of Temporal's durable execution guarantees.
  46. Building durable agents with Temporal and AI SDK by VercelThis post details the integration of Temporal's Durable Execution with the AI SDK by Vercel. It introduces a plugin (`@temporalio/ai-sdk`) that wraps AI SDK calls (like `generateText`) in Temporal Activities, providing automatic retries and state persistence for LLM interactions. It also explains how to implement tool execution within Temporal Workflows by proxying Activities, ensuring that external API calls made by AI agents are also durable. A real-world example of a daily news briefing agent demonstrates how to combine these durable LLM calls and Activity executions for complex, multi-phase AI workflows.
  47. Building observable AI agents: Temporal now integrates with BraintrustThis post details the integration of Temporal with Braintrust to provide observability for AI agents. It introduces code examples for wrapping OpenAI clients with Braintrust's `wrap_openai` and integrating the `BraintrustPlugin` into Temporal Workers and Clients. It also demonstrates how to use `braintrust.load_prompt` to manage prompts externally from Temporal Workflows.
  48. How Goldcast scaled event orchestration to millions using TemporalGoldcast implemented a two-phase Temporal Workflow design for event duplication. Phase 1 involves an atomic and rollback-safe synchronous workflow for core transactional operations, ensuring proper cleanup on failure. Phase 2 is an asynchronous workflow for background tasks like notifications, integrations, and QA checks. This separation isolates critical paths, improves user experience by providing immediate confirmation, and allows for independent retries of background failures. For email delivery, Goldcast uses Temporal Workflows to break down large recipient lists into parallel batches, with a sequential first batch for pagination metadata and parallel processing for remaining batches. Activities return structured error objects for easier failure management, and partial successes are preserved. This has led to a 99.9% success rate for large email sends and significantly reduced processing times.
  49. The hero’s journey to AI durability with TemporalThis post details how to apply Temporal's Durable Execution to AI features to improve reliability. It explains how to prevent duplicate model calls by executing them as Activities within Workflows, manage human-in-the-loop feedback using Signals and Queries, and make AI agent tools more robust by wrapping them in Activities. It also highlights the importance of using Workflow timers for durable waits and contrasts them with standard `asyncio.sleep` calls. The post references a Learn tutorial for a practical walkthrough of building durable AI applications and MCP tools.
  50. The journey to shippable AI systems: Patterns that workThis post introduces five patterns for building shippable AI systems that leverage durable execution: 1. Guarded tool calls (timeouts, retries, circuit breakers, fallbacks, cost metadata). 2. Idempotent side-effects (idempotency keys, compensating actions). 3. Human checkpoints (real approval steps, reviewer timeouts). 4. Deterministic plans, versioned prompts. 5. Cost and policy budgets. It frames these patterns as solutions to common operational issues in AI workflows such as overhead, long-running complexity, and brittle recovery, and reinforces the role of durable execution in providing state, retries, visibility, and long-running task management for these AI systems.
  51. Of course you can build dynamic AI agents with TemporalThis post clarifies that Temporal's deterministic Workflow requirement does not preclude building dynamic AI agents. It explains that the separation of deterministic Workflows (orchestration) and non-deterministic Activities (LLM calls, tool invocations) is key. The post details how Temporal's Durable Execution allows AI agents to make runtime decisions based on LLM outcomes, ensuring that if a failure occurs, the agent can resume from its last consistent state without re-executing LLM calls or diverging from its original path. It provides code examples and architectural diagrams to illustrate this concept, highlighting how Temporal enables non-deterministic agents with durable execution.
  52. Here's how to build durable AI agents with Pydantic and TemporalIntroduces native Temporal support in Pydantic AI, allowing AI agents to leverage Temporal's durable execution for fault tolerance, long-running interactions, and state preservation. This integration automatically wraps non-deterministic agent work (LLM calls, tool executions) as Temporal Activities with built-in retries, while the agent's coordination logic runs as a deterministic Temporal Workflow. Demonstrates this with a two-agent system for a Slack dinner recommendation bot, showcasing orchestration, handling asynchronous messages with Signals, coordinating API calls as Activities, and implementing human-in-the-loop approval flows.
  53. How to think about agentic solutions for the enterpriseThis post highlights the critical gap between AI experimentation and production-ready enterprise AI systems, emphasizing the need for durable execution. It details how current AI frameworks often lack the reliability, observability, and state management required for long-running, complex workflows. The post argues that Temporal's Durable Execution provides the missing backbone for enterprise AI by enabling automatic state tracking, intelligent failure retries, crash recovery, and complete visibility, contrasting this with the limitations of existing agent frameworks and memory systems. It cites case studies of companies like Replit, ZoomInfo, and Gorgias that have successfully transitioned to durable orchestration for their AI applications.
  54. Building a persistent conversational AI chatbot with TemporalThis post details the application of Temporal's Durable Execution to build a persistent, infinitely scalable conversational AI chatbot. It contrasts this approach with traditional stateless architectures, highlighting how treating each conversation as a Temporal Workflow overcomes limitations like session loss, limited scalability, and fragile error handling. The post outlines functional and non-functional requirements for such a chatbot, presents a traditional design with its drawbacks, and then illustrates a Temporal-based design where each conversation is a Workflow. It emphasizes how this enables stateless application proxies, resilient workflows that automatically resume after failures, maintainable modular design through Activities interacting with LLMs and tools, rich context management by passing conversation history to the LLM, and transparent state inspection. The implementation walkthrough focuses on the `ChatbotService.java` class responsible for processing user messages and interacting with Workflows.
  55. From prototype to production-ready agentic AI solution: A use case from Grid DynamicsThis post details Grid Dynamics' migration of a deep research agent from a LangGraph/Redis/Kafka architecture to Temporal. Key contributions include: demonstrating Temporal's ability to manage complex agent state directly within workflows, eliminating the need for external state stores like Redis; showcasing the simplification of error handling and retry logic through declarative `RetryPolicy` configurations, reducing custom code; and highlighting how Temporal's durable execution addresses challenges with exactly-once processing, stale state, and graceful resumption after long pauses, leading to a more scalable and maintainable solution.
  56. How Cargo engineers a smarter go-to-market process with TemporalThis post details Cargo's implementation of event-driven GTM workflows, data synchronization (ETL), and AI workflow orchestration using Temporal. It emphasizes the shift from treating Workflows as job queues to lightweight state machines, pushing heavier data into Activities for improved performance. A code example demonstrates a Temporal Workflow that runs an orchestration 'run' by iterating through an execution graph via `executeNode` Activities, handling failures with `finishRun`, and durably persisting progress.
  57. Building long-running interactive MCP tools with TemporalThis post details the architecture and implementation of long-running, interactive MCP tools powered by Temporal Workflows. It showcases the Temporal Invoice MCP sample, demonstrating how to expose Workflow interactions (start, status, approve, reject) as MCP tools. Key technical aspects include using `client.start_workflow` to initiate Workflows, `handle.signal` for state changes (e.g., `ApproveInvoice`), and `handle.query` for retrieving Workflow state (`GetInvoiceStatus`). The post highlights the use of `workflow.wait_condition` for human interaction and the benefits of Temporal's durable execution for handling complex, multi-step AI processes.
  58. The fallacy of the graph: Why your next agentic workflow should be code, not a diagramThis post argues against using graph-based frameworks for complex, procedural logic, especially for agentic applications. It contrasts the limitations of static graphs in handling dynamic control flow, data management, and error handling/compensations with the benefits of writing normal, procedural code orchestrated by Durable Execution. The author uses examples to show how code-based approaches with try/except/finally and idempotent activities provide cleaner, more robust, and easier-to-maintain solutions for dynamic workflows, including agentic systems and saga patterns, compared to graph-based representations. It reinforces the core principle of Durable Execution as a superior abstraction for such logic.
  59. The unbreakable web: From fragile scripts to bulletproof WorkflowsThis post details the integration of Browserbase's AI-powered browser automation SDK (Stagehand) with Temporal's Durable Execution platform. It demonstrates how to build resilient web automation workflows by encapsulating browser interactions (e.g., clicking buttons, filling forms, extracting data) within Temporal Activities. Key technical contributions include: 1. Using Stagehand's semantic interaction capabilities (e.g., `await stagehand.page.act("click the submit button")`) to make scripts resilient to website HTML/CSS changes. 2. Leveraging Temporal's state persistence and retry policies to ensure workflow completion despite network failures or browser crashes. 3. Designing Activities to be atomic and idempotent for safe automatic retries, illustrated with examples like `initializeBrowser` and `cleanupBrowser`. 4. Showcasing a practical example of a resilient search workflow with specific retry configurations for susceptible Activities like `extractSearchResults`. The post emphasizes the architectural pattern for scaling to use cases like large-scale data collection, fintech automation, and AI-agent orchestration.
  60. Enabling platform engineering with Temporal: 5 practical use casesThis post details five practical use cases of Temporal for platform engineering teams: incident response and runbook automation, certificate rotation, infrastructure management, CI/CD pipeline orchestration, and Internal Developer Platform (IDP) automation. It explains how Temporal Workflows encode logic for these operations, providing fault tolerance, state persistence, human input integration, auditability, and real-time visibility. Specific examples include Datadog's use of Temporal for migration operations, Temporal Cloud's guidance on certificate rotation, and Netflix's re-platforming of Spinnaker. The post emphasizes Temporal's ability to manage complex, multi-step processes reliably, turning operations into stateful, observable, and retryable workflows.
  61. Running GitHub Actions through Temporal: A complete guideThis post details how to use Temporal Workflows and Activities to orchestrate GitHub Actions. It covers GitHub App authentication, a workaround for dispatch ID tracking by injecting a unique identifier into action inputs and searching step names, and the design of a Temporal Workflow to trigger, discover, and monitor GitHub Actions. It also outlines the implementation of Temporal Activities for core GitHub API operations like triggering actions, finding action IDs, and getting action status, including retry logic and heartbeating for long-running jobs.
  62. Production-ready agents with the OpenAI Agents SDK + TemporalThis post details the integration of the OpenAI Agents SDK with Temporal's Durable Execution. It explains how AI agents, defined using primitives like Agents and Handoffs in the OpenAI SDK, can be run as Temporal Workflows. This allows agents to leverage Temporal's capabilities for reliable execution, including handling LLM rate limits, network issues, and application crashes. The integration allows developers to code the happy path for agents, with Temporal managing error handling and ensuring crash-proof execution. The post provides examples of how to define agents and orchestrate them within Temporal Workflows, highlighting the benefits of combining the rapid agent development of the OpenAI SDK with the production-readiness of Temporal.
  63. How Grepsr uses Temporal to deliver scalable and reliable web dataGrepsr adopted Temporal to orchestrate their web data extraction workflows, replacing a brittle system of cron jobs, custom scripts, and message queues. They specifically leveraged Temporal's code-first workflows, built-in fault tolerance for resuming failed jobs, native parallelism for handling thousands of concurrent workflows, and observability features (Web UI, Event History) for debugging and monitoring. This resulted in 99% delivery reliability, a 60% reduction in incident resolution time, faster time-to-market, and higher throughput.
  64. Building production-ready generative AI: How Temporal supercharges Google's Gemini and VeoThis post details how Temporal's Durable Execution platform is applied to build production-ready generative AI systems, specifically using Google's Gemini and Veo models for video generation. It highlights how Temporal addresses common challenges in agentic systems, such as stateful long-running processes, transient failures, costly re-runs, and visibility/debugging. The post provides a high-level architecture of a video generation system and code snippets demonstrating the use of Temporal Workflows and Activities to orchestrate calls to Gemini for scene generation and Veo for video clip creation. It emphasizes the benefits of structured output from LLMs and Temporal's automatic retry mechanisms for external API calls.
  65. Durable Execution meets AI: Why Temporal is ideal for AI agents & Generative AI AppsThis post extends the understanding of Temporal's Durable Execution by demonstrating its applicability to AI agents and Generative AI applications. It maps common AI development concepts (chains, LLMs, tools, memory, checkpointing, human-in-the-loop, long-running processes) to Temporal's core features (Workflows, Activities, Signals, Updates, Queries). The post highlights how Temporal's resilience and state management capabilities inherently address the distributed system challenges of AI applications, such as handling numerous remote requests, transient failures, and complex orchestration, without requiring developers to write explicit retry or checkpointing logic.
  66. Durable MCP: Using Temporal to give agentic systems superpowersThis post introduces the concept of implementing Model Context Protocol (MCP) tools as Temporal Workflows to provide durability, visibility, reliability, and scalability to agentic systems. It demonstrates how to create an MCP tool that executes a Temporal Workflow, which in turn executes a durable Activity. This approach allows agentic systems to leverage Temporal's fault-tolerant execution for tools, ensuring they work every time and can handle enterprise-scale requirements.
  67. From AI hype to durable reality — why agentic flows need distributed-systems disciplineThis post details how to integrate Temporal with AI agentic flows and the Model Context Protocol (MCP). It demonstrates wrapping MCP tool invocations as Temporal Activities, leveraging Temporal's Durable Execution for resilience, automatic retries, timeouts, and state persistence. The post also discusses the application of polyglot workflows for cross-language AI orchestration, the role of Temporal Nexus in unifying AI and business workflows with end-to-end observability, and the use of Signals and Queries to maintain stateful conversations even through failures. It emphasizes how Temporal's features reduce boilerplate code and enhance developer velocity in building AI systems.
  68. How Dust Builds Agentic AI with Temporal WorkflowsThis post details how Dust leverages Temporal Workflows as the core orchestration engine for its AI agent platform. It explains how events from various tools (Slack, Notion, GitHub) trigger Temporal Workflows that handle data ingestion, transformation, AI processing, and indexing. It highlights the use of Temporal for Dust's Tracker product, which monitors documents and triggers agent updates. The post emphasizes that Temporal replaces custom logic for retries, timeouts, and state recovery, allowing Dust to focus on product development. It also mentions the scale of operations (10M+ activities/day on Temporal Cloud) and the use of Datadog for observability. A specific example of filtering files within a Temporal Activity for the GitHub connector is provided, illustrating Temporal's retry and resume capabilities.
  69. The definitive guide to Durable ExecutionThis post provides a foundational definition and explanation of Durable Execution, framing it as 'crash-proof execution'. It elaborates on the concept by detailing four key characteristics: execution virtualization, no time limitation, automatic state preservation, and hardware agnosticism. The post aims to clarify what Durable Execution is and why it's important for building reliable applications, distinguishing it from typical workflow engines.
  70. How many Activities should I use in my Temporal Workflow?This post provides detailed guidance on designing Temporal Activities, focusing on the trade-offs between using many fine-grained Activities versus fewer coarse-grained ones. It outlines specific scenarios where splitting work into multiple Activities is necessary (e.g., routing to different task queues for specialized hardware or security, distinct retry/timeout policies) or practical (e.g., multiple database transactions, external state-changing API calls, cost optimization, dynamic execution, detailed observability). The core recommendation is to aim for one operation per Activity to enhance maintainability, observability, and reliability, while providing clear reasoning for exceptions.
  71. Temporal: Beyond State Machines for Reliable Distributed ApplicationsThis post elaborates on Temporal's Durable Execution model by contrasting it with traditional state machines. It explains how Temporal's event-sourced, durable state tracking and automatic retries allow developers to write synchronous-looking code that executes reliably to completion, abstracting away the complexities of distributed system plumbing like queuing, messaging, and explicit state management. It highlights the benefits of this approach for orchestration, state management, retries, timeouts, distributed reliability, and visibility compared to ad-hoc solutions or explicit state machine implementations.
  72. Trading Airflow + EMR for Temporal + Bauplan: The Mediaset taleThis post details Mediaset's migration from a legacy Airflow + EMR stack to Temporal and Bauplan. It highlights the challenges of the previous setup, including slow data refresh rates and complex infrastructure. The migration to Temporal for orchestration and Bauplan for serverless data pipelines resulted in a near real-time dashboard with a 5-minute refresh rate, reduced development cycles from 3 months to 6 weeks, and simplified infrastructure from over 6 AWS services to just 2 tools. The post emphasizes Temporal's advantages over Airflow, such as durable, replayable workflows with automatic retries and a code-first approach, and Bauplan's serverless data management capabilities.
  73. Designing Robust Financial Ledgers with TemporalThis post details how Temporal's durable execution model can be applied to building robust financial ledgers, ensuring transaction integrity, automatic retries, guaranteed execution for reconciliation, scalability for high-volume transaction processing, end-to-end visibility for auditing and compliance, and flexible workflow orchestration. It highlights the combination of a high-speed transaction layer (e.g., TigerBeetle), a metadata-rich accounting layer (e.g., Formance), and Temporal as the durability layer.
  74. How Layer.ai is upgrading mobile game designLayer.ai integrated Temporal into their architecture to handle heavy asynchronous tasks like file pre-processing, model inference, and post-processing. This decoupled these tasks from their main servers, preventing overload and improving performance. Temporal's built-in retries and observability provided better control over task execution, reducing failures and ensuring reliability. An example 'FileUploadWorkflow' demonstrates the use of Temporal for handling file uploads, including retries, timeouts, and checks to ensure task execution continuity even if steps fail. This architecture allows Layer.ai to scale and process complex workflows more reliably.
  75. Temporal in SpaceThis post details an experiment where a Temporal workflow was run on a Raspberry Pi attached to a weather balloon launched into space. The workflow consisted of a Python script that invoked an Activity every five seconds to read data from a USB sensor, parse it, and store the processed information in Temporal's Event History. The experiment demonstrated Temporal's ability to recover from connectivity loss in the upper atmosphere and seamlessly continue operations once connectivity was restored, highlighting its fault-tolerance capabilities in an extreme environment. The post also includes a link to the GitHub repository for the experiment.
  76. Saga Pattern in Microservices: A Mastery GuideThis post elaborates on the Saga pattern for managing distributed transactions in microservices, a core concept within Temporal's durable execution capabilities. It details the mechanics of Sagas, including choreography vs. orchestration, participants, compensation actions, and error handling. The post emphasizes the importance of idempotency, designing for failure, and monitoring. It positions Temporal as a platform that facilitates the implementation of Sagas, referencing existing case studies and discussing future trends like event-driven architectures and improved data isolation. The core contribution is a deep dive into the Saga pattern's principles and its practical application, directly aligning with Temporal's durable execution offering.
  77. Simplifying distributed transactions with microservicesThis post introduces Temporal's Durable Execution as a solution for simplifying distributed transactions in microservices. It contrasts conventional approaches like stateful services and event stores with Temporal's Workflow abstraction, highlighting how Workflows automatically persist state, provide visibility through Event History, and enable customizable retries. It details the key components: Workflow IDs for unique and single execution, Activities for encapsulating fallible operations with built-in retries, and the Event History for debugging and audit trails. The post emphasizes that Temporal centralizes orchestration logic, leading to more reliable and maintainable systems.
  78. Resilient Event-Driven Architecture for FinanceThis post contrasts choreography-based Event-Driven Architecture (EDA) with Temporal's durable execution paradigm. It highlights the problems with choreography-based EDA, such as scattered business logic, hidden coupling, complex error handling, and limited observability. It then explains how Temporal's durable execution addresses these issues by centralizing business logic in Workflows, providing stateful management, integrated error handling, and unified observability. Best practices for implementing EDA with Temporal are also provided, including treating Workflows as code, modular Activities, error recovery and compensation, dynamic workflow design, and incremental adoption.
  79. Reliable data processing: Queues and WorkflowsThis post contrasts Temporal's Durable Execution with traditional persistent queues. It explains how Temporal Workflows inherently handle challenges like message loss, retry logic, dead-letter queues, ordering, and message duplication, which are significant operational burdens for queue-based systems. The post highlights that Temporal Workflows provide crash-proof execution, automatically persist state, and offer built-in mechanisms for handling failures and retries, thereby reducing the development and maintenance overhead associated with implementing these patterns manually.
  80. How Temporal Builds Durable Cloud Control SystemsThis post details how Temporal Cloud's control plane leverages Temporal's durable execution model for critical operations like namespace provisioning and rolling upgrades. It explains the cell-based architecture of the data plane and how the control plane uses workflows to manage complex, long-running tasks, including entity workflows for resource lifecycle management. The post highlights the benefits of this approach for developer productivity and reliability in building managed cloud services.
  81. Fault Tolerance in Distributed Systems | Reliable WorkflowsThis post elaborates on the fundamental principles of fault tolerance in distributed systems, such as redundancy, replication, failover mechanisms, and graceful degradation. It explicitly connects these concepts to Temporal's core offering, highlighting how durable workflows, automatic retries, and state management directly implement these fault tolerance strategies. The post reinforces that Temporal simplifies building resilient applications by handling state preservation and task retries, reducing the burden on developers to implement custom error-handling logic.
  82. Simplifying Scalability in Distributed Systems with Workflow OrchestrationThis post elaborates on how Temporal's durable execution model simplifies the management of state and retries in distributed systems. It highlights how Temporal abstracts away the complexities of sagas and state machines, allowing developers to focus on business logic while the platform handles state persistence, automatic retries, and recovery from interruptions. The post uses Bugcrowd's transition to microservices and Temporal as a case study, demonstrating a 2x faster engagement time, 400% capacity increase, and 15 hours saved weekly due to reduced manual processes.
  83. Salesforce streamlines Workflows, cuts code by 95%Salesforce's adoption of Temporal as a foundational technology for stitching together complex, acquired systems. They highlight a 95% code reduction for certain workflows by leveraging Temporal's primitives, moving from disconnected automation scripts and manual processes to a unified orchestration layer. The post emphasizes the value of Temporal for long-running workflows and improved developer productivity through unit-testable code. Salesforce is also building an ecosystem of tools around Temporal for self-service capabilities and auto-debugging to support broader adoption.
  84. Snakes, chaos, and the resilience of Temporal: A live demo breakdownThis post demonstrates the resilience of Temporal's Durable Execution model through a live demo of a multiplayer snake game. It highlights how Temporal Workflows orchestrated game state, player snakes, and game rounds, and how the system recovered from induced worker failures (via apple consumption) and unexpected node failures (via cable pulling). The demo showcased Temporal's ability to persist state and seamlessly resume execution after disruptions, mimicking real-world production scenarios. It also touched upon the use of Socket.io for low-latency UI integration and the performance benefits of the Go SDK for worker recovery activities.
  85. Event-Driven Systems & Loosely Coupled Architectures ExplainedThis post explains how Temporal's durable execution and version patching features address the challenges of evolving event structures in event-driven architectures. It highlights how Temporal allows workflows to adapt to changes in event API surfaces and event protocol versions, ensuring reliable continuation of long-running processes. It also details how workflows can be replayed from the start without duplicating work if versioned event contracts are not supported, provided determinism is maintained.
  86. How BPMN and legacy orchestration tools are holding you backThis post argues against the use of BPMN and legacy orchestration tools due to their limitations in scalability and development speed. It contrasts these limitations with the benefits of a code-first, scalable, and reliable approach, exemplified by Temporal. The post details how Temporal abstracts error handling and state management, enabling faster project delivery and developer innovation. It also includes a case study of ANZ Bank's successful migration to Temporal, highlighting an 8x improvement in project delivery speed.
  87. How ZoomInfo Built a Scalable Cluster with Temporal CloudThis post details how ZoomInfo adopted Temporal Cloud for their dynamic audience management system, handling hundreds of millions of operations monthly. It highlights the challenges of their previous homegrown solution, including throughput limitations and batch processing difficulties. The adoption of Temporal Cloud led to significant reductions in development time (from weeks to days) and debugging efforts (from daily to near-maintenance mode). The migration to Temporal Cloud from a self-hosted cluster resulted in cost savings. The post also showcases the use of Temporal's long-running workflows for their new Copilot product with Account AI, processing large volumes of data and scaling effectively. Key benefits emphasized include workflow-as-code, built-in observability, and the cloud-agnostic nature of Temporal.
  88. Multi-agent Workflow patterns and modern use cases | TemporalThis post details how Temporal's durable execution capabilities are applied to orchestrate and coordinate multi-agent workflows. It explains how Temporal acts as a conductor for interactions among agents, manages stateful coordination and context sharing, enables reliable asynchronous communication, supports parallel processing and concurrency control, and handles long-running workflows with multi-step processes. It also covers event-driven triggers, signals, and real-time agent coordination, illustrating these concepts with a customer support example. The post emphasizes that Temporal doesn't require building specific code for each agentic framework, acting as a general-purpose backbone for agentic systems.
  89. Events are the wrong abstraction: Rethinking distributed systemsThis post argues that Durable Execution, as exemplified by Temporal, is a superior abstraction to Event-Driven Architecture (EDA) for building distributed systems. It details common problems with EDA, such as lack of APIs, tight coupling, scattered logic, poor debuggability, ad-hoc state machines, increased latency, race conditions, and system upgradability issues. It then presents Durable Execution as an alternative that centralizes business logic, manages state consistently, survives crashes gracefully, and designs processes flexibly. The post uses a payment accumulation system as an example to illustrate the difference in architectural diagrams between EDA and Temporal.
  90. How to handle fault tolerance in microservicesThis post contributes to the Durable Execution feature thread by providing a comparative analysis of fault tolerance strategies in microservices. It contrasts RPC (highlighting its limitations in partial failures and client burden), persistent queues (discussing ordering issues and dead-letter queues), and workflows. It positions Temporal's workflow-as-code approach as a superior solution for built-in resilience, long-running processes, and enhanced visibility, directly addressing the challenges of managing failures in distributed systems.
  91. Temporal Boosts Video Processing by 600% at The Washington PostThis post details how The Washington Post's Arc XP division used Temporal to significantly improve their video processing efficiency, achieving a 600% runtime improvement in their automated thumbnail selection system. They migrated from a legacy internal orchestration system to Temporal, leveraging its durable execution and multi-language support (Python, Java, TypeScript, Golang) to distribute workloads across inexpensive AWS instances. The article highlights the ease of debugging with Temporal's Replay feature, contrasting it with their previous system's complexity. The team also noted the benefit of Temporal's ability to handle asynchronous distributed execution and the inspiration drawn from new features like Temporal Nexus.
  92. Spooky Stories: The vanishing messages at Snap towerThis post details Snap's real-world challenges with managing ephemeral messaging in a high-volume microservices architecture, specifically addressing issues with message loss, delivery failures, and inefficient scaling of traditional message queues. It highlights how Temporal's durable execution capabilities were adopted to create resilient systems that automatically manage retries, track message states, and ensure data integrity even during system disruptions, thereby replacing unreliable message queues.
  93. Spooky Stories: ​​Taming deployment complexity with TemporalThis post demonstrates how Temporal Workflows can be used to manage complex deployment processes, specifically detailing the "7 Levels of Hell Deployment." It illustrates how Workflows can abstract multi-repository builds (Level 2), handle blue-green deployments by orchestrating infrastructure updates and router configuration (Level 3), and incorporate manual approval steps via integrations like Slack (Level 4). The post highlights how Temporal's durable execution, retries, and state persistence simplify these previously complex "systems" into manageable Workflow definitions.
  94. Modernizing monoliths with TemporalThis post details a recommended approach for modernizing monolithic applications by leveraging Temporal Workflows to manage complex, cross-domain processes. It introduces the concept of 'domain thinking' and 'ephemeral state' in the context of monoliths and explains how Temporal Workflows provide a durable and observable solution for orchestrating these processes. The post provides a code example of a Temporal Workflow for order management, highlighting the benefits of code-first orchestration, process visibility, and simplified testing. It also discusses how Temporal can add durability and visibility to existing monoliths without requiring a full microservices decomposition.
  95. Spooky Stories: Chilling Temporal anti-patterns (part 2)This post provides guidance on Temporal best practices through a series of anti-patterns, framed with horror movie analogies. It details how to avoid 'mega workflows' by breaking them into smaller, manageable workflows, leveraging Nexus for inter-team process management. It advocates for using Temporal's native state management over external state machines to prevent 'split-brain' scenarios. The post also advises on error handling, recommending the use of default infinite retries for activities unless business logic strictly requires limited retries, and emphasizes the use of compensation patterns (Saga pattern) to ensure successful outcomes even when intermediate steps fail. Finally, it highlights the importance of Temporal's observability features (SDK metrics, Cloud metrics, Web UI, Search Attributes) for debugging and understanding workflow execution.
  96. Mastering Durable Execution in Distributed SystemsThis post expands on the concept of durable execution by detailing new programming possibilities it unlocks. It provides concrete examples of writing code that can sleep for months, receive RPCs and respond to queries using Signals and Queries, and store state directly in instance variables instead of a database. It also reiterates how durable execution simplifies distributed system design patterns like event-driven architecture, task queues, and sagas by providing automatic orchestration and a superior developer experience compared to traditional approaches.
  97. Spooky Stories: Money transfers on the edge of townThis post details the migration of a financial institution's SFTP-based daily sweep process to Temporal. The original system suffered from frequent SFTP upload/download failures, SFTP server overload, and database query failures under load, impacting internal system resilience and deadline adherence. The migration to Temporal addressed these issues by leveraging its durable execution capabilities, enabling incremental rollout, and utilizing event history for error investigation. The outcome was immediate improvements in stability and resilience.
  98. Durable RAG with Temporal and ChainlitThis post details how to integrate Temporal with Retrieval-Augmented Generation (RAG) and Chainlit to build Durable RAG systems. It explains how Temporal's durable execution model can be used to orchestrate complex, long-running AI tasks involved in RAG, ensuring reliability and scalability. The post highlights Temporal's role in managing workflow orchestration, scalability, and resource optimization for RAG models, and how Chainlit enhances user interaction with these systems.
  99. Building Resilient Sweep Networks with Temporal WorkflowsThis post details the incremental migration of a financial institution's sweep network system to Temporal, focusing on stabilizing SFTP operations and integrating human-in-the-loop approvals. The use of Temporal Activities for SFTP uploads/downloads and the automatic retry mechanism significantly improved the system's resilience to intermittent failures. The incremental migration strategy, leveraging existing Go code and Temporal's message passing (Signals and Queries), reduced risk and accelerated stability improvements. The event history provides an enhanced audit trail for debugging and operational insights.
  100. What it means to be a Durable Execution Service ProviderThis post defines and elaborates on the concept of "Durable Execution" as the core abstraction provided by Temporal. It explains that Temporal enables any engineering team to become a "Durable Execution Service Provider" by leveraging its SDK and supervisor service. The supervisor service (Temporal Server or Temporal Cloud) ensures that applications built with the SDK can resume execution after failures, guaranteeing exactly-once completion. The post highlights that Durable Execution is the key differentiator from previous workflow engines and positions Temporal as the market leader in this abstraction.
  101. Managing very long-running Workflows with TemporalThis post introduces the concept of 'Entity Workflows' for managing very long-running or indefinite processes. It details the 'Continue-As-New' mechanism as a solution to Temporal's Event History limits (50K Events/50MB) for these workflows, explaining its mechanics and considerations such as input design and signal draining. It also briefly mentions the interaction patterns for long-running workflows via Signals and Queries, and the necessity of Workflow Versioning for updates.
  102. Building application reliability on top of infrastructure unreliabilityThis post explains how Temporal's Durable Execution directly addresses the challenge of building reliable applications on top of unreliable infrastructure. It contrasts the limitations of traditional microservice architectures and load balancers in handling failures with Temporal's approach, where workflows automatically resume on another process after interruptions. The post uses an e-commerce example to illustrate how Temporal Workflows can maintain state and continue execution from the point of failure, thereby providing application-level reliability and answering critical operational questions about request status and failure points.
  103. Community threads: Is it possible to create an event-based Workflow?This post explains how to build event-based workflows in Temporal by leveraging `wait_condition` and signals. It demonstrates how to use signals to advance workflow stages and how `wait_condition` can pause workflow execution until a specific condition is met. The example code, using the Python SDK, illustrates a workflow that executes a series of activities, waiting for stage advancements via signals between them. This pattern ensures that workflows can reliably handle external events and acknowledgments, even over extended periods and in the face of worker failures, due to Temporal's replay functionality.
  104. Nuon gets 36x increase in developer velocity with Temporal long-lived WorkflowsNuon rebuilt their API using Temporal long-lived Workflows, migrating from traditional event-sourcing patterns. This resulted in a 36x increase in developer velocity, achieving in two sprints what took 36 developer months previously. The Temporal UI is now used for tracking object lifecycles in the database, providing detailed views of creation and updates. The post highlights how Temporal Workflows manage database objects and interactions, creating queues per object for efficient and scalable data management.
  105. The distributed machineThis post introduces the concept of a 'distributed machine' enabled by Temporal's durable execution. It draws parallels between the evolution of computing machines (physical, virtual, containers) and how Temporal abstracts distributed processing. The core contribution is framing Temporal not just as a tool for reliability, but as a new paradigm for executing code across distributed systems as if it were a single, higher-level machine, obviating the need for application-specific distributed systems.
  106. Parallelism and ConcurrencyThis post clarifies the fundamental concepts of concurrency and parallelism, defining them as the ability to handle multiple things in an undefined order and the ability for multiple things to run literally at the same time, respectively. It explains how these concepts apply to single-threaded environments (like JavaScript) through event loops and to multi-core/multi-machine environments. The post then details how Temporal's Workflow execution model, with its deterministic event loop, provides both concurrency and parallelism, along with fault tolerance, for distributed systems. It uses an online food ordering application as an example to illustrate how Temporal Workflows manage complex, stateful processes that benefit from these properties.
  107. What is time-travel debugging?This post introduces and explains the concept of time-travel debugging, also known as reverse debugging, detailing its history and implementation techniques (Record & Replay, Snapshotting, Instrumentation). It specifically highlights how Temporal's durable execution leverages a form of record & replay to enable time-travel debugging of production code. The post explains that durable execution records external inputs and state changes, allowing for deterministic replay of workflow executions on development machines. This provides a powerful debugging capability that goes beyond traditional debuggers and distributed tracing, allowing engineers to inspect any past state of a production workflow.
  108. Customer Workflows in Loyalty Programs with TemporalThis post demonstrates how to implement the Actor Model's core principles (message sending/receiving, state maintenance, actor creation) using Temporal Workflows. It provides a concrete example of a customer loyalty program, detailing the implementation of Signal handlers for 'addPoints' and 'cancelAccount', and the use of `workflow.NewSelector` to manage incoming messages. It also discusses the implications of event history size for long-lived workflows and the use of Continue-As-New.
  109. Saga Orchestration vs ChoreographyThis post introduces the Saga pattern and contrasts choreography with orchestration. It explains how Temporal's durable execution inherently provides orchestration capabilities, avoiding the single point of failure common in traditional orchestration engines. The core contribution is demonstrating how Temporal's code-based workflow logic and state persistence enable reliable and failure-resilient implementation of the Saga pattern, driving towards completion of all steps without manual intervention.
  110. 25 key terms for speaking Distributed Systems and TemporalThis post defines and explains 25 key terms related to distributed systems and Temporal, including 'durable execution', 'concurrency', 'scalability', 'reliability', 'eventual consistency', 'strong consistency', 'CAP Theorem', 'ACID', 'atomicity', 'isolation', 'durability', 'idempotent function', 'deterministic function', 'platform', 'application', 'event sourcing', 'Temporal', 'Worker', 'Workflow', 'Activity', 'Query', 'Signal', 'retry', 'Cluster', and 'History'. It aims to provide a foundational understanding for conversations around these topics.
  111. Workflows as Actors: Is it really possible?This post details how Temporal Workflows can be used to implement the Actor Model. It explains the core tenets of the Actor Model (message passing, creating other actors, maintaining state) and maps them to Temporal's features: Signals and Updates for message passing, Child Workflows for creating other actors, and the inherent statefulness of Workflows for maintaining state. It also discusses the use of Queries for retrieving state and the implications of long-running Workflows.
  112. Saga Design Pattern Explained for Distributed SystemsThis post explains the Saga design pattern and how Temporal's Durable Execution inherently supports it. It details how Temporal automatically handles state saving, retries, and recovery from failures, allowing developers to focus on implementing compensation logic. The post contrasts Sagas with Two-Phase Commit (2PC) and provides pseudocode and code examples in Java and Go to illustrate the implementation of compensation activities within Temporal Workflows.
  113. Saga Compensating TransactionsThis post introduces and details the implementation of the compensating action pattern for distributed systems, specifically within the context of Temporal Workflows. It explains the problem of maintaining data consistency across microservices and presents compensating actions as a solution to undo state changes when a sequence of operations fails. The post provides code examples for implementing this pattern in Java (using the Saga library), Python, and TypeScript, highlighting the importance of correct ordering of compensation steps and handling potential failures during compensation execution. It also references the Saga pattern as a broader framework that utilizes compensating actions.
  114. Getting started with small-step operational semanticsThis post introduces Small-Step Operational Semantics (SOS) as a formal method to specify and reason about the execution of sagas. It designs a mini-language, SagaLang, to illustrate SOS for basic sagas, sagas with retries (forward recovery), and sagas with compensations (backward recovery), highlighting the need for idempotence and compensation guarantees for true transaction-like behavior.
  115. How Durable Execution worksThis post elaborates on the mechanics of Temporal's Durable Execution by detailing the client-server-worker architecture and the sequence of events involved in workflow execution. It explains how the Temporal Server stores the event history, how workers poll for tasks (workflow tasks and activity tasks), and how the server orchestrates retries for failed activities. The post uses a food delivery example to illustrate the flow from starting a workflow, calling an activity, and handling activity completion or failure, emphasizing the role of the event history in ensuring state consistency and durability.
  116. Building Distributed Systems in NodeThis post introduces the concept of durable execution and demonstrates its implementation using Temporal's TypeScript/JavaScript SDK. It explains how durable execution simplifies distributed systems by persisting state and automatically retrying operations, even across process failures. The post provides a practical example of a food delivery app, showcasing how to create orders, charge customers, and retrieve order status using Temporal Workflows, Signals, and Queries. It details the code for starting a workflow, handling potential errors with retries and application failures, and defining queries for retrieving workflow state. The post emphasizes that durable execution allows developers to write code that can run indefinitely and interact with long-running processes without concern for transient failures.
  117. The linear developer experienceThis post introduces the concept of a "tipping point" in software complexity, where simple requirements can trigger an explosion of complexity and a non-linear deterioration of the developer experience. It argues that traditional frameworks like Django are not designed for complex, long-running, or multi-transaction processes, leading to "fear-driven architecture" and "Rube Goldberg Doctor Seuss machines." The post advocates for a "linear developer experience" provided by platforms like Temporal, where workflow orchestration is treated as a "distributed systems as a service," abstracting away persistence, retries, and timing. This allows developers to focus on business logic and maintain a readable sequence of steps, avoiding the negative consequences of tipping points.
  118. Do... or do not: Why Yoda doesn't use microservicesThis post introduces the concept of a dedicated state management platform for microservices, drawing parallels to atomic database transactions. It argues that microservices, while beneficial for development cycles, introduce significant state management challenges. The author proposes that a platform like Temporal, which provides durable execution for workflows, abstracts away these complexities. This allows developers to achieve exactly-once execution guarantees for their microservice interactions, similar to ACID transactions in databases, without needing to manually manage state during failures or retries. The post highlights how this abstraction simplifies development and improves reliability by ensuring that workflows run to completion, even in the face of system disruptions.
  119. Paper summary: End-to-end arguments in system designThis post applies the End-to-End Argument from Saltzer, Reed, and Clark to system design within the context of Temporal. It uses the example of reliable file transfer to illustrate how critical functions like failure detection and mitigation should be primarily implemented at the application layer for completeness and correctness, while the platform layer can provide optimizations. This principle guides the placement of functionality in Temporal systems to ensure robustness and reliability.
  120. Node.js Task Queue SolutionThis post introduces Temporal as a powerful alternative to traditional Node.js task queues like Bull, Bee Queue, Agenda, and Kue. It demonstrates how Temporal's Workflows and Activities can be used to implement similar functionality, such as sending emails, with added benefits. Key contributions include: - A direct comparison of setup between a traditional task queue (Bull) and Temporal for sending emails. - Highlighting Temporal's advantages in UI monitoring, type safety, built-in retry policies with exponential backoff, and the use of `sleep()` for delays. - Showcasing how Temporal Workflows can manage multi-part tasks and conditional logic, enabling complex sequences of operations like a customer welcome email series. - Demonstrating the use of Signals for conditional execution within a workflow.
  121. Caching API Calls with Long-Lived Workflows in TemporalThis post demonstrates a practical application of Temporal's durable execution for API call caching using long-lived workflows. It introduces the pattern of using a `while(true)` loop with `sleep` for scheduled API refreshes and `continueAsNew` to manage workflow state and avoid event limits. It also shows how to use Activities for external API calls and Queries for retrieving cached data, providing concrete code examples in TypeScript.
  122. Introduction to Temporal WorkflowsThis post introduces Temporal Workflows as a core abstraction for reliable and scalable distributed systems, contrasting them with regular function executions. It highlights how Workflows simplify the development of long-running, stateful processes by providing a developer experience equivalent to database transactions, mitigating failure at the platform level and guaranteeing correctness at the application level. The post uses a monthly reminder email use case to illustrate the complexities of implementing such logic with regular functions (requiring cron jobs, message queues, and manual failure/retry handling) versus the simplicity and robustness offered by Temporal Workflows, which allow for direct translation of pseudocode into reliable execution.
  123. Inversion of ExecutionThis post introduces and explains the core architectural concept of 'Inversion of Execution' in Temporal. It details how the Temporal Server orchestrates workflow and activity tasks, which are then executed by client-side Workers. This model is contrasted with traditional execution models where a server directly runs application code. The post elaborates on the benefits of this approach, including enhanced reliability through deterministic replay of workflow tasks, improved security via zero-trust and encrypted payloads, and simplified scalability by separating server and worker concerns. A bank transfer example illustrates the workflow and activity task execution flow.
  124. Tips on defining WorkflowsThis post introduces fundamental concepts of defining workflows, emphasizing iterative improvement, domain specificity, and robust failure handling. It contrasts Domain-Specific Languages (DSLs) with general-purpose programming languages, detailing the pros and cons of flowchart, form, and markup-based DSLs, and noting their limitations with complex workflow definitions. The core idea of workflows as a structured set of objectives to achieve a goal is central.
  125. What does long running mean?This post redefines the concept of 'long running' in the context of Temporal, shifting the focus from arbitrary time cutoffs to handling uncertainty in distributed systems. It explains that Temporal is suitable for both very short, transactional operations (like file updates) and infinitely long processes (like background checks that can take days). The key takeaway is that Temporal's durable execution, event history persistence, and the ContinueAsNew API allow developers to model entities and entire customer journeys reliably, regardless of the actual execution time. It also contrasts Temporal's low latency for fast responses with the higher latency of traditional workflow engines and emphasizes the importance of monitoring ScheduleToStart latency.
  126. Media processing Workflow solutionThis post introduces a media processing workflow solution built with Temporal. It details the problem of orchestrating complex, stateful, and fault-tolerant workflows, specifically for media processing involving IoT sensor data. The solution outlines the business requirements and maps them to Temporal Workflows and Activities. Key technical aspects covered include: defining Activity Options with `StartToCloseTimeout` and `RetryPolicy` for managing activity execution and failures; using `workflow.CreateSession` to schedule related activities on the same worker instance; and demonstrating how to handle conditional logic (e.g., terminating a workflow early if media is not obtainable) and iterative processing with retries. The post provides code examples in Go for the workflow, activities, and a simulated vendor API, illustrating the practical application of Temporal's durable execution for robust media processing pipelines.
  127. Workflow Engine Design Principles with TemporalThis post provides a foundational explanation of workflow engine design principles, detailing how Temporal addresses core requirements such as task queues, timers, consistency, scalability, sharding, system workflows, and external implementations. It highlights the critical importance of transactional updates across all components for maintaining system consistency and avoiding race conditions. The post also outlines Temporal's scalability strategy, which focuses on scaling out the number of workflows rather than scaling up individual workflow instances, and discusses the role of sharding in partitioning databases and hosts for large-scale systems.
  128. Dealing with failureThis post discusses three patterns for dealing with failure: Request-Reply (RPC), Persistent Queues, and Workflows. It highlights the limitations of Request-Reply in handling partial failures and the complexity it pushes to the client. Persistent Queues offer automatic retries and decoupling but are less suitable for request-response scenarios and require handling "poison messages." Workflows are presented as a robust solution for complex, long-running processes, offering intelligent failure handling, statefulness, and individual addressability, while inheriting some challenges from queues like response correlation and "poison message" handling. The post advocates for choosing the appropriate pattern based on tradeoffs for specific scenarios.
  129. A journey: Durable Task Framework, Uber, & open source magicThis post chronicles the author's journey and the evolution of ideas leading to Temporal, starting from AWS SWF, moving to Azure Durable Task Framework (DTFx) and its successor Azure Durable Functions, and then to Uber's Cadence. It highlights the recurring problem of developers spending excessive time building resiliency with low-level primitives and the motivation to create a more developer-friendly, stateful workflow system. The post introduces the concept of "workflow-as-code" and the use of async/await in C# for building such systems. It also touches upon the creation of open-source projects like Cherami and Cadence at Uber to address large-scale messaging and stateful application needs, culminating in the founding of Temporal Technologies.
  130. Deep Dive into Temporal Workflow stress testingThis post details two stress testing scenarios, 'rabbit' and 'reactor', designed to validate Temporal's performance and reliability under high load. The 'rabbit' scenario focuses on exponential load generation through nested child workflows and tests completion, termination, and cancellation propagation. The 'reactor' scenario simulates sustained load with long-running workflows processing external signals. The post also highlights three specific bugs discovered and fixed as a result of these tests: broken parent-to-child workflow cancellation in the Go SDK, nil pointer exceptions during cancels/terminates with high fan-out, and failure of cancellation/termination propagation after a child workflow continued-as-new.