TechPrufer
Get started
Two Sigma
Two Sigma logoTwo Sigma · Feature trail

Fair Preemptive Resource Scheduler

1 post · Nov 2016

What this trail covers

Two Sigma developed Cook, a fair preemptive resource scheduler for compute clusters. Cook addresses the challenge of balancing latency-sensitive interactive jobs with throughput-oriented batch jobs by using a Cumulative Resource Share (CRS) metric to prioritize tasks and preempt lower-priority tasks when necessary. Benchmarks show that preemption significantly reduces latency for interactive jobs with a minimal impact on overall cluster utilization.

All Two Sigma trails
  1. Cook: A Fair Preemptive Resource Scheduler for Compute ClustersThis post introduces Cook, a fair preemptive resource scheduler for compute clusters. It details the problem of balancing interactive and batch job requirements, proposes a Cumulative Resource Share (CRS) metric for fairness and prioritization, and explains how preemption is used to reclaim resources for higher-priority tasks. Benchmarks using a production trace demonstrate that Cook effectively reduces latency for interactive jobs while maintaining high cluster utilization.