Failure Handling Patterns

5/21/2026

This post introduces a pattern for building human-in-the-loop approval workflows using Temporal. It addresses common failure modes in document approval processes such as unanswered requests, missed deadlines, and lost state across restarts. The solution leverages Temporal's durable execution to ensure decisions are waited on reliably, SLAs are enforced automatically, escalations occur without external schedulers, and all actions are recorded.

5/11/2026

Introduces the Temporal Validated Pattern for handling permanent workflow failures. This pattern allows workflows to pause on specific error types (e.g., invalid input, policy breaches) and resume after external correction, avoiding full restarts. It leverages queryable metadata for routing blocked processes to resolution resources and provides operators with visibility into individual execution states and pipeline-wide status.

1/20/2026

This post details Goldcast's implementation of Temporal Workflows for two key use cases: 1) Reliable email delivery at scale, involving breaking down large recipient lists into parallel batches, with a focus on resilient error handling and performance optimizations. 2) Event duplication orchestration using a two-phase workflow design: a synchronous, rollback-safe main workflow for core transactional operations and an asynchronous workflow for background processing of non-critical tasks. The post provides code examples and metrics demonstrating improved success rates and reduced processing times.

1/8/2025

This post provides a conceptual overview of fault tolerance in distributed systems, defining it as the ability of a system to keep operating during failures. It contrasts fault tolerance with high availability, outlines key components like redundancy and failover, and discusses factors to consider such as cost and performance trade-offs. It highlights how cloud and microservices architectures necessitate application-level resilience, and positions Temporal as a solution that simplifies this by preserving workflow state and automatically retrying failed tasks. The post also lists best practices for designing fault-tolerant systems and provides real-world examples, concluding with Temporal's specific features for fault tolerance like state management and automatic retries.

10/31/2024

This post uses a fictional narrative to illustrate the real-world challenges Snap faced with ephemeral messaging in a microservices architecture, specifically concerning message delivery failures and the limitations of traditional message queues. It then explains how Snap adopted Temporal to solve these issues by leveraging its durable execution model for automatic retries, state tracking, and reliable recovery from disruptions, ensuring no data is lost.

10/17/2024

This post details best practices for idempotency and local activities, emphasizing the need for idempotency keys to prevent duplicate side effects, especially with local activities. It also explains Temporal Reset as a 'time travel' mechanism for fixing bugs in production by rewinding and replaying workflows. The post warns about workflow history size limits and suggests using Continue-As-New to manage it. It stresses the importance of deterministic workflows and introduces workflow versioning and patching for safe code updates. Finally, it encourages developers to leverage existing Temporal features like Signals, Queries, and Updates, and to test for determinism.

6/27/2024

This post details the migration of a financial institution's sweep network system to Temporal. It focuses on stabilizing SFTP operations and database-intensive tasks by wrapping them in Temporal Activities, leveraging Temporal's automatic retry mechanism for transient failures. The post outlines a phased migration approach, starting with SFTP operations, and describes how Temporal Workflows manage the orchestration, while Activities handle external interactions like SFTP uploads/downloads and database queries. It also details the integration with a legacy system, using Signals and Queries to coordinate the Temporal Workflow's execution with the existing process, and highlights the benefit of Temporal's event history for auditing and debugging.

2/27/2024

This post elaborates on idempotency as a critical property for durable systems like Temporal. It explains the concept of idempotency with real-world examples and details how Temporal's 'at-least-once' execution guarantee for Activities necessitates idempotent design. The post provides practical implementation patterns for building idempotent Activities, including using idempotency keys derived from workflow and activity IDs, and checking for pre-existing results. It also includes a SQL example demonstrating how to prevent duplicate side effects using an operations table with a uniqueness constraint and discusses potential race conditions and mitigation strategies.

5/2/2023

This post provides detailed code examples for implementing the compensating action pattern (also known as compensating transactions) in Java, Python, TypeScript, and Go. It demonstrates how to manage compensating actions within Workflows to ensure data consistency in distributed systems when operations fail. The post emphasizes the correct ordering of compensations and the need for them to be idempotent, handling cases where the original operation may or may not have completed.

11/8/2022

This post introduces best practices for handling failures within Temporal Workflows, categorizing errors by 'where' (platform vs. application) and 'when' (transient, intermittent, permanent). It provides guidance on configuring retry policies with exponential backoff, handling non-retryable errors, and implementing backward recovery strategies using pseudocode for undoing critical process steps. The post also introduces the concept of `NewNonRetryableApplicationError` for explicitly marking errors that should not be retried.

6/22/2021

This post details the four types of Activity timeouts in Temporal: Schedule-To-Close, Start-To-Close, Heartbeat, and Schedule-To-Start. It explains the purpose of each timeout, provides code examples for setting them in Go and Java, and illustrates their lifecycle within the Temporal system. The post emphasizes the importance of Start-To-Close for preventing stuck Activity executions and Schedule-To-Close for managing retries. It also highlights the use of Heartbeat timeouts for long-running Activities and suggests monitoring metrics for Schedule-To-Start instead of explicit timeouts.

1/21/2021

This post introduces and analyzes three distinct patterns for handling failures in distributed systems: Request-Reply (RPC), Persistent Queues, and Workflows. For each pattern, it elaborates on its mechanics, advantages (e.g., simplicity, automatic retries, robust partial failure handling), and disadvantages (e.g., client-side complexity, queue clogging, response correlation challenges). The post uses examples like financial transactions and cloud orchestration to illustrate the practical implications of these patterns and concludes that no single pattern is universally optimal, advocating for a pragmatic approach that combines patterns based on specific system requirements.

11/19/2020

This post introduces principles for designing effective error messages for developers. It emphasizes writing errors for users, suggesting causes and fixes, making them searchable, logging them appropriately, and avoiding excessive verbosity. It also provides a tactical style guide for error messages, including using standard Go error packages, lowercase and no punctuation, actionable messages, and specific phrasing like 'unable to...'.

10/21/2020

This post details the author's personal journey and contributions to the development of stateful orchestration systems, including AWS SWF, Azure Durable Task Framework (DTFx), Uber's Cherami and Cadence, and ultimately Temporal. It emphasizes the recurring problem of developers spending significant effort on building resiliency and how these systems aim to abstract that complexity. The post highlights the use of async/await in C# for DTFx and Golang for Cadence, and the open-source nature of Temporal.

9/10/2020

Introduces two stress testing scenarios ('rabbit' and 'reactor') for Temporal. The 'rabbit' scenario tests Workflow tree spawning, completion, termination, and cancellation with varying fan-out and depth. The 'reactor' scenario tests sustained load from signals on long-running Workflows. Details specific bugs found in cancellation/termination propagation and handling, including race conditions and nil pointer exceptions.

8/21/2020

This post details the completion of the JavaSDK rewrite and its release, compatible with server V0.29.0. The team is now fully focused on stabilization. It also elaborates on the stabilization efforts, shifting the perspective to 'Code completion for tests' and 'Identifying and fixing problems discovered by tests'. The team is evaluating Gremlin for failure injection to test cluster and worker operations under various infrastructure failures. Testing on Java has commenced in parallel with the SDK development, covering new and previously untested paths.