BlogsTemporalWorker Resource Auto-Tuning

Worker Resource Auto-Tuning

Worker Resource Auto-Tuning

4
posts
2023–2026

Worker auto-tuning has been introduced to simplify management by automatically adjusting available Task Slots based on CPU and Memory utilization targets. This feature aims to prevent out-of-memory issues and handle bursts of low-resource-usage activities. Users can configure target memory and CPU usage, or optionally set minimum/maximum slots and ramp throttle times for more granular control. Static slot allocation is also an alternative. Future work includes auto-tuning of pollers. This post provides specific examples of how EvenUp uses `temporal_worker_task_slots_used`, `temporal_workflow_task_schedule_to_start_latency`, and `temporal_activity_schedule_to_start_latency` metrics for slot-based and latency-based scaling. It also details the implementation of a Kubernetes liveness probe using `temporal_num_pollers`, `temporal_worker_task_slots_available`, and `temporal_request{operation="RespondWorkflowTaskCompleted"}` to automatically restart unhealthy workers, addressing issues like stuck workers due to synchronous CPU-intensive work in Workflow tasks and failed Workflow cache evictions.

2026

When running your own orchestrator isn’t worth it anymore — and why EvenUp chose Temporal

3/17/2026

This post details how EvenUp migrated their existing workloads to Temporal. They specifically highlight their use of Temporal's metrics for scaling workers, employing both slot-based scaling using `temporal_worker_task_slots_used` and latency-based fallbacks with `temporal_workflow_task_schedule_to_start_latency` and `temporal_activity_schedule_to_start_latency`. Additionally, they describe the implementation of a Kubernetes liveness probe using metrics like `temporal_num_pollers`, `temporal_worker_task_slots_available`, and `temporal_request{operation="RespondWorkflowTaskCompleted"}` to automatically restart unhealthy workers, resolving issues with stuck workers and periodic processing slowdowns.

2024

Resource-based auto-tuning for Temporal Workers

11/12/2024

Introduces resource-based auto-tuning for Temporal Workers, allowing automatic adjustment of Task Slots based on target CPU and memory usage. This feature simplifies worker management by reducing manual right-sizing and aims to prevent out-of-memory issues and handle bursts of low-resource activities. It provides configuration options for target memory/CPU usage, minimum/maximum slots, and ramp throttle times, as well as a static slot allocation option. The feature is currently in pre-release.

Product Updates from Replay 2024

9/19/2024

This post announces the general availability of Worker Auto-Tuning, which automatically adjusts worker slots based on CPU and memory usage. It also introduces Worker Scaling, which uses real-time task queue metrics to inform scaling decisions for worker fleets.

2023

An introduction to Worker tuning

10/23/2023

This post provides foundational knowledge for deploying and managing Worker Pools, focusing on performance tuning Worker-side code. It discusses key metrics like `workflow_task_schedule_to_start_latency` and `worker_task_slots_available` to determine when to scale Workers up or down. It also details configurations like `MaxConcurrentActivityExecutionSize` and `MaxConcurrentWorkflowTaskExecutionSize`.