TechPrufer
Get started
Temporal
Temporal logoTemporal · Feature trail

Temporal Cloud Persistence Layer

3 posts · Mar 2021 – Oct 2024

What this trail covers

Temporal Cloud offers a multi-cloud platform, expanding its infrastructure from AWS to Google Cloud. This expansion involved leveraging Temporal's own capabilities for cloud-agnostic workflows, addressing challenges in containerization, orchestration, and infrastructure-as-code across different cloud providers. The migration involved adapting to provider-specific APIs and resource models, and finding solutions for services like the visibility layer where direct compatibility was lacking. The use of a custom persistence layer with efficient sharding, WAL, and tiered data storage has been shown to significantly reduce application-side latency compared to self-hosted instances, even when accounting for increased network latency.

All Temporal trails
  1. Making Temporal Cloud a Multi-Cloud PlatformThis post details the technical challenges and solutions involved in migrating Temporal Cloud's infrastructure from AWS to Google Cloud. It highlights the limitations of standard containerization and orchestration technologies (like Kubernetes) and Infrastructure-as-Code (IaC) tools when dealing with multi-cloud environments due to provider-specific APIs and resource models. The post provides concrete examples of differing Kubernetes Service annotations and Terraform HCL for load balancers between AWS and GCP. It also discusses the need for custom solutions for services like the visibility layer (AWS OpenSearch vs. GCP equivalent). The core technical contribution is demonstrating how Temporal's own durable execution capabilities were used to manage the complexity of this multi-cloud expansion, ensuring consistency and reliability across different cloud providers.
  2. Cloud Benchmark: Temporal Cloud vs. Self-HostedThis post presents a benchmark comparing application-side latency between Temporal Cloud and self-hosted Temporal instances. It quantifies latency improvements across five SDK metrics (WorkflowEndtoEnd, StartWorkflowExecution, SignalWorkflowExecution, RespondWorkflowTaskCompleted, RespondActivityTaskCompleted), demonstrating that Temporal Cloud offers significantly lower latency. The improvements are attributed to Temporal Cloud's custom persistence layer, which features more efficient sharding, a write-ahead log (WAL), and tiered data storage, designed for high throughput and large scale. The benchmark results suggest that the benefits of this custom persistence layer outweigh the increased network latency incurred when using Temporal Cloud. The post also discusses the operational and cost benefits of Temporal Cloud over self-hosting for production workloads.
  3. Faster Throughput & Lower Latency with Temporal CloudThis post details Temporal Cloud's custom persistence layer, focusing on three key architectural pillars: better sharding for dynamic resizing and handling traffic spikes, a write-ahead log (WAL) to aggregate writes and improve recovery, and tiered storage to move completed Workflow Event Histories to object storage, reducing database load and improving performance. These improvements result in higher scalability and lower latency compared to self-hosted clusters.
  4. Go shadowing + bad choices = our first data loss bug in yearsThis post details a critical data loss bug in Temporal's persistence layer that was caused by a Golang variable shadowing issue. The bug only manifested under specific conditions involving instability in the underlying Cassandra cluster, leading to corrupted workflow states. The post describes the debugging process, the initial misattribution to persistence vendor issues, the eventual root cause identification, and the subsequent poor communication strategy by Temporal. It outlines the steps taken to fix the bug, backport it, and improve future communication and testing practices, including hardening tests for dependency-level failures, adding linters to catch shadowing issues, improving graceful handling of corrupted workflows, and augmenting the version update mechanism to alert users of critical bugs.