Temporal Cloud Persistence Layer

5/22/2024

This post presents a benchmark comparing the application-side latency of Temporal Cloud against a self-hosted Temporal Service. It details the benchmark setup using a Kubernetes cluster, Omes for load testing, and the measurement of five SDK metrics: WorkflowEndtoEnd Latency, StartWorkflowExecution Latency, SignalWorkflowExecution Latency, RespondWorkflowTaskCompleted Latency, and RespondActivityTaskCompleted Latency. The results consistently show lower latency in Temporal Cloud across all measured metrics, attributing these improvements to the custom persistence layer's efficient sharding, WAL, and tiered storage.

3/20/2024

This post details the technical implementation of Temporal Cloud's custom persistence layer. It explains the rationale behind building a custom solution to address multi-tenancy challenges and the 'noisy neighbor' problem in databases. The post outlines three key architectural pillars: better sharding for dynamic scaling of persistent state across multiple databases, a write-ahead log (WAL) to aggregate writes and improve database performance, and tiered storage to move completed Workflow Event Histories to object storage, thereby reducing database load and improving overall efficiency and latency.

3/3/2021

This post details a data loss bug in Temporal's persistence layer caused by Golang variable shadowing, which occurred when Cassandra returned specific errors during cluster instability. It explains the debugging process, the root cause, and the impact on workflow creation. The post also outlines corrective actions including adding tests for dependency-level failures, implementing a linter for variable shadowing, improving graceful handling of corrupted workflows, and enhancing communication mechanisms for critical bugs.