Java CPU Profiling

7/22/2026

This post details the evolution of Java CPU profiling, highlighting the limitations of JFR's ExecutionSample for CPU-bound workloads and the challenges of using unsupported JVM internals like AsyncGetCallTrace. It introduces the new JFR CPUTimeSample event in JDK 25, which, combined with cooperative stack walking, provides a more accurate and stable CPU profiling mechanism by sampling based on actual CPU time rather than just observed thread activity.

9/18/2025

This post details the evolution of Datadog's internal code optimization efforts, moving from manual Go code tuning to an agentic system called BitsEvolve. The manual phase focused on identifying and optimizing performance hotspots in critical, high-throughput services by analyzing assembly code and understanding real-world input distributions, leading to significant speedups and cost savings. The learnings from manual optimization, particularly the need for deep understanding of code behavior and the limitations of manual scaling, informed the development of BitsEvolve. This agentic system leverages evolutionary algorithms, inspired by research like AlphaEvolve, to automatically mutate, evaluate, and iterate on code variants against performance benchmarks, aiming to scale deep optimization work across the organization.

7/17/2025

This post details the investigation and resolution of a memory regression in Go 1.24, specifically related to the Go heap and the `mallocgc` function, which caused increased RSS usage. The investigation involved analyzing system metrics (RSS) versus Go runtime metrics, examining `/proc/[pid]/smaps`, and collaborating with the Go community to pinpoint the root cause. The post also sets up a follow-up on how Go 1.24's Swiss Tables implementation led to memory reductions.

1/31/2023

This post details performance improvements in the Datadog Agent's metrics pipeline, specifically focusing on optimizing the computation of unique keys for metrics. The team identified a bottleneck in the original algorithm's sorting and deduplication of tags. They implemented several strategies: specializing the sorting algorithm based on the number of tags, switching to a faster hash implementation (murmur3), and optimizing map access with 64-bit keys. The most significant improvement came from redesigning the context generation algorithm to use XOR bitwise operations instead of merging hashes, removing the need for sorting. A custom hash set was introduced for efficient tag deduplication, avoiding the overhead of Go maps.

2/28/2022

This post details contributions to Go 1.18's CPU profiler, specifically addressing issues with setitimer(2) and timer_create(2) on Linux to improve CPU burst accuracy on multi-core systems. It also covers bug fixes for profiler labels in Go 1.18, enhancing the ability to associate arbitrary key/value pairs with goroutines for better profiling analysis.

9/30/2021

This post details the use of Datadog's Continuous Profiler to identify a performance bottleneck in an Akka application. The issue was traced to the `ForkJoinPool` within the default Akka dispatcher, which was consuming significant CPU due to frequent thread parking and unparking caused by an irregular flow of tasks from the `LatencyReportActor`. The solution involved reconfiguring the `LatencyReportActor` to use a different dispatcher with a more stable task flow, resulting in a 30% drop in overall CPU usage and a reduction in the default Akka dispatcher's thread pool size.

10/7/2020

This post details the development of Datadog's statistical Python profiler. It explains the rationale for statistical profiling over deterministic profiling (like cProfile) for production environments due to overhead concerns. The architecture is described, including the recorder, collectors (stack, memory, lock), exporter (pprof format), and scheduler. The stack collector's implementation in Cython and its dynamic polling rate adjustment for low overhead are highlighted. The memory and lock collectors are also introduced.