TechPrufer
Get started
Temporal
Temporal logoTemporal · Feature trail

Failure Handling Patterns

17 posts · Aug 2020 – May 2026

What this trail covers

Temporal Workflows are used for reliable email delivery at scale, handling large volumes with smart batching and resilient error handling. Event duplication involves a two-phase approach: an atomic, rollback-safe main workflow for core operations and a subsequent asynchronous workflow for background tasks like notifications and integrations. This has led to significant improvements in success rates, processing times, and reduced manual intervention. The Validate pattern has been introduced for robust error handling and rollback mechanisms in distributed systems, leveraging Temporal's durable execution to guarantee compensation logic runs even if workers crash. This post details the implementation of sagas with compensating transactions, emphasizing the importance of idempotency for retries and the use of timeouts as a first line of defense. It highlights how Temporal's durable execution simplifies complex, long-running processes and provides an effectively exactly-once guarantee for business processes.

All Temporal trails
  1. Human-in-the-Loop Approval WorkflowsThis post details the application of the Temporal Validated Pattern to build a robust human-in-the-loop approval system. It addresses common failure modes in document approval processes, such as unanswered requests, silent deadline misses, lost context across restarts, and incomplete audit trails. The solution leverages Temporal's durable execution to ensure decisions are waited on durably, SLAs are enforced automatically, escalations occur without explicit schedulers, and all actions are recorded reliably, even during infrastructure disruptions.
  2. Recover Failed Workflow Steps Without RestartingThis post introduces the Temporal Validate pattern, a new approach to handling permanent failures in multi-step business processes. Unlike traditional methods that require restarting the entire process, the Validate pattern allows workflows to pause at a permanent failure point (e.g., invalid input, policy breach). It enables human or automated agents to inspect, correct the underlying issue, and then resume the workflow from where it left off without losing progress on already completed steps. The pattern also includes queryable metadata for routing blocked processes to appropriate resolution resources and provides operators with full visibility into the state of individual executions and the entire pipeline.
  3. How Goldcast scaled event orchestration to millions using TemporalThis post details how Goldcast uses Temporal Workflows for large-scale email delivery (100k-200k emails) with a smart batching strategy and resilient error handling, achieving a 99.9% success rate and reducing processing times by 50%. It also describes a two-phase Temporal Workflow design for event duplication: an atomic, rollback-safe main workflow for core operations and a separate asynchronous workflow for background tasks, improving success rates to 99.8% and reducing manual cleanup requests to zero.
  4. Error handling in distributed systems: A guide to resilience patternsThis post details essential resilience patterns for distributed systems, focusing on error handling. It explains partial failures, network unreliability, and asynchronous chaos, introducing Temporal's Durable Execution as a solution for effectively exactly-once guarantees and stateful coordination in long-running processes. The post elaborates on the saga pattern with compensating transactions, providing a code example for implementing rollbacks. It also covers critical patterns like timeouts (connection and request) with smart strategies and retries (exponential backoff with jitter), emphasizing the golden rule of retries: idempotency. The post highlights Temporal's ability to write fault-tolerant code that looks synchronous but executes asynchronously, simplifying complex distributed system development.
  5. Fault Tolerance in Distributed Systems | Reliable WorkflowsThis post provides a conceptual overview of fault tolerance in distributed systems, defining it as the ability of a system to keep operating during failures. It contrasts fault tolerance with high availability, outlines key components like redundancy and failover, and discusses factors to consider such as cost and performance trade-offs. It highlights how cloud and microservices architectures necessitate application-level resilience, and positions Temporal as a solution that simplifies this by preserving workflow state and automatically retrying failed tasks. The post also lists best practices for designing fault-tolerant systems and provides real-world examples, concluding with Temporal's specific features for fault tolerance like state management and automatic retries.
  7. Spooky Stories: The vanishing messages at Snap towerThis post uses a fictional narrative to illustrate the real-world challenges Snap faced with ephemeral messaging in a microservices architecture, specifically concerning message delivery failures and the limitations of traditional message queues. It then explains how Snap adopted Temporal to solve these issues by leveraging its durable execution model for automatic retries, state tracking, and reliable recovery from disruptions, ensuring no data is lost.
  8. Spooky Stories: Chilling Temporal anti-patterns (part 1)This post details best practices for idempotency and local activities, emphasizing the need for idempotency keys to prevent duplicate side effects, especially with local activities. It also explains Temporal Reset as a 'time travel' mechanism for fixing bugs in production by rewinding and replaying workflows. The post warns about workflow history size limits and suggests using Continue-As-New to manage it. It stresses the importance of deterministic workflows and introduces workflow versioning and patching for safe code updates. Finally, it encourages developers to leverage existing Temporal features like Signals, Queries, and Updates, and to test for determinism.
  9. Building Resilient Sweep Networks with Temporal WorkflowsThis post details the migration of a financial institution's sweep network system to Temporal. It focuses on stabilizing SFTP operations and database-intensive tasks by wrapping them in Temporal Activities, leveraging Temporal's automatic retry mechanism for transient failures. The post outlines a phased migration approach, starting with SFTP operations, and describes how Temporal Workflows manage the orchestration, while Activities handle external interactions like SFTP uploads/downloads and database queries. It also details the integration with a legacy system, using Signals and Queries to coordinate the Temporal Workflow's execution with the existing process, and highlights the benefit of Temporal's event history for auditing and debugging.
  10. What is idempotency? And why it matters for durable systemsThis post introduces the concept of idempotency and its importance in durable execution systems like Temporal. It explains how to achieve idempotency in Temporal Activities using idempotency keys (e.g., `workflowRunId + '-' + activityId` or natural keys from external systems) and by checking for pre-existing results. It also provides a SQL example for preventing duplicate side effects and discusses potential race conditions when checking for existing state. The post emphasizes Temporal's 'at least once' execution guarantee for Activities and recommends making Activities idempotent to handle retries safely.
  11. Saga Compensating TransactionsThis post provides detailed code examples for implementing the compensating action pattern (also known as compensating transactions) in Java, Python, TypeScript, and Go. It demonstrates how to manage compensating actions within Workflows to ensure data consistency in distributed systems when operations fail. The post emphasizes the correct ordering of compensations and the need for them to be idempotent, handling cases where the original operation may or may not have completed.
  12. Retry logic in Workflows: Best practices for failure handlingThis post details best practices for handling failures in Temporal Workflows, focusing on the 'where' (platform vs. application) and 'when' (transient, intermittent, permanent) dimensions of failure. It explains how to configure retry policies with exponential backoff for platform-level errors like connection issues, and how to handle application-level errors (e.g., insufficient funds) by marking them as non-retryable and implementing backward recovery. The post also introduces the use of `NewNonRetryableApplicationError` and provides pseudocode for a recovery mechanism.
  13. The four types of Activity timeoutsThis post details the four types of Activity timeouts in Temporal: Schedule-To-Close, Start-To-Close, Heartbeat, and Schedule-To-Start. It explains the purpose of each timeout, provides code examples for setting them in Go and Java, and illustrates their lifecycle within the Temporal system. The post emphasizes the importance of Start-To-Close for preventing stuck Activity executions and Schedule-To-Close for managing retries. It also highlights the use of Heartbeat timeouts for long-running Activities and suggests monitoring metrics for Schedule-To-Start instead of explicit timeouts.
  14. Dealing with failureThis post introduces and analyzes three distinct patterns for handling failures in distributed systems: Request-Reply (RPC), Persistent Queues, and Workflows. For each pattern, it elaborates on its mechanics, advantages (e.g., simplicity, automatic retries, robust partial failure handling), and disadvantages (e.g., client-side complexity, queue clogging, response correlation challenges). The post uses examples like financial transactions and cloud orchestration to illustrate the practical implications of these patterns and concludes that no single pattern is universally optimal, advocating for a pragmatic approach that combines patterns based on specific system requirements.
  15. A journey: Durable Task Framework, Uber, & open source magicThis post details the author's personal journey and contributions to the development of stateful orchestration systems, including AWS SWF, Azure Durable Task Framework (DTFx), Uber's Cherami and Cadence, and ultimately Temporal. It emphasizes the recurring problem of developers spending significant effort on building resiliency and how these systems aim to abstract that complexity. The post highlights the use of async/await in C# for DTFx and Golang for Cadence, and the open-source nature of Temporal.
  16. Deep Dive into Temporal Workflow stress testingIntroduces two stress testing scenarios ('rabbit' and 'reactor') for Temporal. The 'rabbit' scenario tests Workflow tree spawning, completion, termination, and cancellation with varying fan-out and depth. The 'reactor' scenario tests sustained load from signals on long-running Workflows. Details specific bugs found in cancellation/termination propagation and handling, including race conditions and nil pointer exceptions.
  17. Temporal Transparency Update #6This post details the completion of the JavaSDK rewrite and its release, compatible with server V0.29.0. The team is now fully focused on stabilization. It also elaborates on the stabilization efforts, shifting the perspective to 'Code completion for tests' and 'Identifying and fixing problems discovered by tests'. The team is evaluating Gremlin for failure injection to test cluster and worker operations under various infrastructure failures. Testing on Java has commenced in parallel with the SDK development, covering new and previously untested paths.