Blogs›Datadog›Kubernetes Job System Performance Optimization

Kubernetes Job System Performance Optimization

posts

2021

Datadog migrated its job system to Kubernetes, initially experiencing a significant performance regression with jobs completing at a 40-50% slower rate and higher CPU utilization. Through detailed performance experimentation, including careful metric selection (focusing on idle CPU and throughput over load average), the team identified and addressed overheads. Key optimizations involved tuning Kubernetes resource requests (CPU and memory) to improve pod scheduling density, aiming for six pods per node. Analysis of per-pod overhead, including `containerd-shim` CPU and memory usage, revealed it to be relatively small (10ms/pod/second CPU, 1-5MB memory). Further investigation into work execution patterns, such as analyzing `mpstat` output, helped pinpoint CPU utilization issues. The post details the iterative process of experimentation, measurement, and tuning to minimize Kubernetes overhead and restore performance parity with the previous VM-based system.

2021

How we minimized the overhead of Kubernetes in our job system | Datadog

2/22/2021

This post details the process of minimizing overhead for a job system migrated to Kubernetes. It covers: 1. Designing performance experiments with comparable VM and Kubernetes setups. 2. Selecting appropriate performance metrics, specifically idle CPU and job completion throughput, and explaining why load average was misleading. 3. Tuning Kubernetes resource requests (CPU and memory) to achieve higher pod density per node. 4. Analyzing per-pod overhead by measuring `containerd-shim` CPU and memory usage. 5. Investigating work execution patterns using tools like `mpstat` to identify CPU utilization issues. The core contribution is the systematic approach to diagnosing and resolving performance regressions introduced by Kubernetes orchestration.