Kubernetes Job System Performance Optimization

Datadog migrated its job system to Kubernetes, initially experiencing a significant performance regression with jobs completing at a 40-50% slower rate and higher CPU utilization. Through detailed performance experimentation, including careful metric selection (focusing on idle CPU and throughput over load average), the team identified and addressed overheads. Key optimizations involved tuning Kubernetes resource requests (CPU and memory) to improve pod scheduling density, aiming for six pods per node.