Java CPU Profiling

8 posts · Oct 2020 – Sep 2026

What this trail covers

Datadog's APM profilers have evolved to provide unbiased CPU and async task profiling. For Java, this involved collaborating with OpenJDK to introduce a new CPU profiling event. For Python, a new async-aware profiling model reconstructs relationships between asyncio tasks, addressing limitations of traditional flame graphs and reducing profiler overhead. This aims to provide more accurate and actionable performance insights for complex asynchronous applications.

All Datadog trails

Sep 2026How we built an async-aware Python profiler | DatadogThis post details the development of an async-aware Python profiler. It explains how asyncio's execution model (coroutines, tasks, event loop) differs from synchronous code, leading to flattened flame graphs that obscure task relationships. The solution reconstructs these relationships by tracking parent-child task dependencies and building a task dependency tree to create 'stacked stacks'. It also addresses edge cases like tasks outliving their creators and discusses overhead reduction techniques, achieving over 60% reduction.
Jul 2026Unbiased Java CPU profiling with JFR in JDK 25 | DatadogThis post details the collaboration with SAP, Amazon, and the OpenJDK community to introduce the `CPUTimeSample` event in JDK 25. This new event, combined with cooperative stack walking, provides accurate CPU-time sampling for Java applications, addressing the bias and incompleteness issues previously encountered with JFR's `ExecutionSample` and unsupported JVM internals like `AsyncGetCallTrace`. The post explains the mechanics of profilers, the limitations of existing JVM sampling, and the trade-offs involved in achieving accurate CPU profiling.
Sep 2025From hand-tuned Go to self-optimizing code: Building BitsEvolve | DatadogThis post details the evolution of Datadog's internal code optimization efforts, moving from manual Go code tuning to an agentic system called BitsEvolve. The manual phase focused on identifying and optimizing performance hotspots in critical, high-throughput services by analyzing assembly code and understanding real-world input distributions, leading to significant speedups and cost savings. The learnings from manual optimization, particularly the need for deep understanding of code behavior and the limitations of manual scaling, informed the development of BitsEvolve. This agentic system leverages evolutionary algorithms, inspired by research like AlphaEvolve, to automatically mutate, evaluate, and iterate on code variants against performance benchmarks, aiming to scale deep optimization work across the organization.
Jul 2025How we tracked down a Go 1.24 memory regression across hundreds of pods | DatadogThis post details the investigation and resolution of a memory regression in Go 1.24, specifically related to the Go heap and the `mallocgc` function, which caused increased RSS usage. The investigation involved analyzing system metrics (RSS) versus Go runtime metrics, examining `/proc/[pid]/smaps`, and collaborating with the Go community to pinpoint the root cause. The post also sets up a follow-up on how Go 1.24's Swiss Tables implementation led to memory reductions.
Apr 2024.NET Continuous Profiler: Exception and lock contention | DatadogThis post details the implementation of exception and lock contention profiling within Datadog's .NET continuous profiler. For exceptions, it describes fetching details like type, message, and location using `ICorProfilerCallback::ExceptionThrown` and `ICorProfilerInfo::GetClassFromObject`. For lock contention, it explains how CLR events like `ContentionStart` and `ContentionStop` are captured, and how duration is computed, including recent contributions to the .NET runtime for richer `ContentionStart` event data. It also outlines strategies for older .NET versions using EventPipe and ETW.
Advertisement
Feb 2024.NET Continuous Profiler: CPU and wall time profiling | DatadogThis post details the implementation of CPU and wall time profiling in the Datadog .NET continuous profiler. It explains how application threads are monitored and sampled, including handling of managed and native threads. The implementation of CPU profiling involves sampling runnable threads and measuring their CPU consumption using OS-specific APIs (NtQueryInformationThread on Windows, /proc/self/task/<tid>/stat on Linux). The post also highlights an internal example where the CPU profiler was used to reduce CPU consumption by identifying a memory-intensive code path in the profiler itself.
Jan 2024.NET Continuous Profiler: Under the hood | DatadogThis post details the technical architecture and implementation choices behind Datadog's .NET profiler, focusing on its continuous profiling capabilities. It explains the profiler's architecture, including samplers, providers, aggregators, and exporters, and how it generates .pprof files. It also covers backend processing, including associating profiles with traces and spans using runtime IDs, and the techniques used to clean up .NET call stacks for improved readability. The implementation details highlight the use of native code (C++ and Rust) and its integration as an ICorProfilerCallback plugin during application startup.
Jan 2023Performance improvements in the Datadog Agent metrics pipeline | DatadogThis post details performance improvements in the Datadog Agent's metrics pipeline, specifically focusing on optimizing the computation of unique keys for metrics. The team identified a bottleneck in the original algorithm's sorting and deduplication of tags. They implemented several strategies: specializing the sorting algorithm based on the number of tags, switching to a faster hash implementation (murmur3), and optimizing map access with 64-bit keys. The most significant improvement came from redesigning the context generation algorithm to use XOR bitwise operations instead of merging hashes, removing the need for sorting. A custom hash set was introduced for efficient tag deduplication, avoiding the overhead of Go maps.
Feb 2022Profiling improvements in Go 1.18 | DatadogThis post details contributions to Go 1.18's CPU profiler, specifically addressing issues with setitimer(2) and timer_create(2) on Linux to improve CPU burst accuracy on multi-core systems. It also covers bug fixes for profiler labels in Go 1.18, enhancing the ability to associate arbitrary key/value pairs with goroutines for better profiling analysis.
Sep 2021How we optimized our Akka application using Datadog’s Continuous Profiler | DatadogThis post details the use of Datadog's Continuous Profiler to identify a performance bottleneck in an Akka application. The issue was traced to the `ForkJoinPool` within the default Akka dispatcher, which was consuming significant CPU due to frequent thread parking and unparking caused by an irregular flow of tasks from the `LatencyReportActor`. The solution involved reconfiguring the `LatencyReportActor` to use a different dispatcher with a more stable task flow, resulting in a 30% drop in overall CPU usage and a reduction in the default Akka dispatcher's thread pool size.
Oct 2020How we wrote a Python profiler | DatadogDatadog built a statistical profiler for Python to address the limitations of deterministic profilers like cProfile for production environments. The profiler is designed with minimal overhead and simple deployment, comprising a recorder, collectors (stack, memory, lock), an exporter using the pprof format, and a scheduler. The stack collector, partially written in C with Cython, gathers execution stack information for Python threads and dynamically adjusts its polling rate to manage CPU overhead. Memory and lock collectors are also included to track allocations and lock contention.