Datadog Feature Trails

Datadog's profiling capabilities have expanded to include Python. This post details the development of a statistical profiler for Python, addressing the limitations of deterministic profilers like cProfile for production environments. The Python profiler is designed with low overhead, simple deployment, and cross-platform compatibility. It comprises a recorder, collectors (stack, memory, lock), an exporter (using pprof format), and a scheduler. The stack collector, written partly in Cython for performance, gathers execution stacks of Python threads, dynamically adjusting its polling rate to minimize overhead. The memory and lock collectors provide insights into allocation and contention issues. The pprof format is used for efficient data export.

Datadog built a system to measure data completeness across all ingestion pipelines in real-time for every customer. The system tracks payloads end-to-end by splitting pipelines into segments and counting creates and acknowledgments for unique payload identifiers within time buckets. This allows for localization of completeness degradation and aggregation into an end-to-end view, handling complex pipeline structures and edge cases like delayed data and retries. The Stream Router system, responsible for data aggregation and post-processing, is a key component. The company also employs best practices for building highly reliable batch data pipelines, including using isolated, short-lived clusters (often on spot instances) for Spark jobs, breaking down long-running jobs vertically and horizontally, and implementing robust monitoring with cluster-level tagging. This approach ensures fault tolerance and efficient recovery from failures.

Datadog built a managed data replication platform to deliver highly reliable, highly scalable, and flexible data movement across the company. This platform abstracts operational overhead, provides robust monitoring and alerting, and adapts to new use cases. It evolved from addressing performance issues in a shared Postgres database by rerouting search queries to a dedicated search platform with dynamically denormalized data. The provisioning process was automated using Temporal workflows, breaki This post details the re-architecture of Datadog's Kubernetes-based PostgreSQL clusters to ensure safe and automatic failover. The previous architecture, which separated leader and read replica pools and used asynchronous replication for standbys, prioritized availability over durability during network partitions, leading to stale data and unsafe failover scenarios. The new architecture leverages synchronous replication for failover candidates, coordinated by Patroni and ZooKeeper, to guarantee data consistency during failover. This involves reconfiguring Patroni to use synchronous replication for standby nodes that are candidates for promotion, ensuring that a standby is sufficiently up-to-date before it can be promoted. The post also discusses the trade-offs between durability and latency, and the validation of the approach through benchmarking and failure testing.

Datadog's BewAIre system has evolved from detecting malicious code in pull requests to scanning entire dependency packages and upstream package registries. This expansion leverages a two-stage evaluation pipeline: a fast, inexpensive filter phase using a previous-generation or high-speed LLM, followed by a more thorough agentic investigation phase for flagged changes. The investigation phase utilizes tools to gather additional context, such as GitHub API calls for commit history, contributor analysis, and code scanning. This post details the initial build of the LLM-powered system for detecting malicious pull requests in real-time, focusing on prompt engineering, data tuning, and handling context window limitations. The system achieved >99.3% accuracy on a curated dataset and has been running in production across Datadog's repositories.

Datadog introduced a steganography-based system to embed widget metadata within screenshots. This system allows for the invisible encoding of a Redis cache key into the pixel data of a widget's border. When a screenshot is taken and pasted, the embedded key can be used to retrieve the original widget's metadata from Redis, enabling features like live previews and context preservation even when users opt for screenshots over direct sharing links. The implementation involves compressing and encoding metadata into an 8-byte key, which is then hidden in the widget's 1px border by subtly altering pixel colors. This watermark is designed to be resilient to various image manipulations and display variations. The system scales to over a billion watermarks per day by leveraging Redis caching and organization-specific keys to prevent collisions.

Datadog built a real-world evaluation platform for its autonomous SRE agents (Bits Investigation). This platform addresses the challenge of subtle regressions in agent behavior by enabling reproducible investigation environments. It consists of a curated label set defining representative investigations (ground truth root cause + world snapshot of signals) and an orchestration platform to execute and score agent performance against these labels. Label creation is scaled by embedding it into the product, using customer feedback and investigation data to construct new labels. The platform tracks performance over time and across different model/configuration variants, allowing for consistent measurement and detection of regressions.

This post details the debugging and resolution of a performance issue in Datadog's Postgres database related to an upsert query for tracking host ingestion times. The problem stemmed from unexpected Write-Ahead Logging (WAL) activity and increased disk writes, even when upserts resulted in no actual data modification. The solution involved analyzing WAL records using pg_walinspect and optimizing the query to leverage Heap-Only (HOT) updates by ensuring updates only modified unindexed columns and by setting an appropriate fillfactor for the table. This significantly reduced write amplification and WAL syncs, improving overall database performance.

Datadog's MCP (Model Context Protocol) server has evolved significantly to provide an observability interface specifically designed for AI agents. Initial versions were simple API wrappers, but the system has been re-architected to address agent constraints like context window efficiency, data retrieval limitations, and tool management. Key developments include optimizing data formats (CSV, YAML over JSON), implementing token-budget-based pagination, introducing query languages (SQL) for data analysis, and enhancing the system to facilitate writing postmortems by integrating structured metadata from Datadog's Incident Management app with unstructured discussions from related Slack channels. This integration uses an ensemble of LLM models to generate postmortem first drafts, with a focus on limiting hallucinations, increasing accuracy, and generating output more quickly through refined LLM instructions and an experimentation framework. Safeguarding mechanisms include prioritizing information based on context, lowering model temperature, and adjusting LLM parameters. An adjacent project for LLM-generated incident summaries also informed this work.

Datadog's Go Agent binaries have undergone significant size reductions, decreasing by up to 77% through systematic dependency auditing, targeted refactors, and re-enabling linker optimizations. This effort involved analyzing build tag usage, isolating dependencies into separate packages, and leveraging tools like `go list` and `goda` for dependency graphing. The project also led to contributions to the Go toolchain and improvements for other large Go projects like Kubernetes. The Datadog Lambda Extension was re-engineered in Rust, resulting in an 82% reduction in cold start latency, a 40% reduction in memory usage, and a binary size reduction from 55 MB to 7 MB. This rewrite focused on minimizing overhead in the resource-constrained Lambda environment, including manually writing AWS API calls to avoid SDK performance penalties and implementing flexible flush strategies (end of invocation, race, or periodic) to balance telemetry immediacy with CPU and transfer costs.

Datadog developed an eBPF-based File Integrity Monitoring (FIM) system to address the limitations of traditional methods like periodic scans, inotify, and auditd. The system ingests billions of file-related kernel events per minute and employs a multi-stage filtering approach. Initially, agent-side rules discard irrelevant events, reducing the volume to approximately one million events per minute. Further optimization involves moving significant event evaluation logic into eBPF programs to pre-filter events. This post details hardening eBPF for runtime security, focusing on lessons learned from Datadog Workload Protection. It covers challenges in program loading and kernel hook points across different kernel versions and distributions, including compatibility issues with program types, helpers, maps, hook point availability, function inlining, and verifier sensitivity. It also discusses capturing and enriching data correctly, monitoring and auditing eBPF usage, operating alongside other eBPF users, measuring and controlling performance cost, and safe rollout practices.

Datadog is evolving its approach to reliability by prioritizing graceful degradation over preventing all failures. This involves implementing persistent intake storage to prevent data loss, making live data available faster by skipping backlogs and prioritizing critical telemetry, and updating retry logic to avoid overwhelming downstream systems. The company is also addressing architectural bottlenecks and technical debt, and introducing prioritization at the infrastructure and compute level to handle network latency issues by investigating and resolving multiple hidden bottlenecks, including Envoy CPU throttling and a Linux kernel bug affecting network adapter performance.

Datadog's Husky event store has evolved its query engine to handle interactive querying at scale, processing over 100 trillion events and billions of queries daily. The engine is designed to manage queries across diverse and schema-less event data, spanning petabytes across millions of object store fragments. This post details the evolution of Husky's underlying data storage layer, focusing on efficient compaction strategies to manage fragments. Compaction involves merging small fragments into larger ones, and the system has been re-architected to separate compute from storage for independent scaling of ingestion, storage, and query paths, offering greater flexibility in isolation, performance, and quality of service in multi-tenant environments. The third-generation system aims to support arbitrary dimensional aggregates at query time, long-term retention of critical data, and querying/aggregation on any field without pre-indexing.

Datadog's live process and container metrics pipeline has been significantly optimized. Initially, it collected data from all hosts in a tenant's infrastructure whenever a user viewed the Processes or Containers pages, leading to millions of data points per second. The system has evolved to collect real-time data only from hosts actively being viewed by the user (up to 50 hosts), drastically reducing data volume. This was achieved by filtering data based on 'host subscriptions' and propagating this state to the intake service via Kafka. Sorting logic was also updated to use standard 10-second interval data instead of high-frequency 2-second data, simplifying the system and improving efficiency. These changes resulted in a 100x reduction in real-time traffic volume, a 98% decrease in infrastructure costs, and lower Datadog Agent resource utilization.

Datadog has evolved its real-time timeseries storage engine for the sixth generation, building a new system in Rust. This iteration focuses on high throughput and low latency, aiming to address challenges posed by increasing data volume, complexity, and cardinality. The new engine is designed to handle high-cardinality workloads, complex queries, and bursty traffic patterns, achieving significant performance gains in ingestion and query speed. The architecture continues to separate real-time data storage (RTDB) from the index database, with RTDB nodes comprising intake, storage engine, snapshot, query execution, and throttling subsystems. This generation represents a significant architectural shift from previous iterations which used Cassandra, Redis, MDBM, Go-based B+ trees, and RocksDB for distribution metrics.

Datadog's Go Agent binaries have undergone significant size reductions, decreasing by up to 77% through systematic dependency auditing, targeted refactors, and re-enabling linker optimizations. This effort involved analyzing build tag usage, isolating dependencies into separate packages, and leveraging tools like `go list` and `goda` for dependency graphing. The project also led to contributions to the Go toolchain and improvements for other large Go projects like Kubernetes. This post details how Go 1.24's Swiss Tables implementation, a new hash table design, significantly reduced memory usage in large in-memory maps by optimizing bucket structure and reducing overhead, leading to fleet-wide savings.

Datadog developed and open-sourced dtln-rs, a real-time, client-side noise suppression library based on the Dual-Signal Transformation LSTM Network (DTLN). This library is designed to be embeddable in native clients and web applications, offering high performance without server dependencies. It leverages WebAssembly, Node.js native modules, and native Rust targets to integrate with WebRTC, enabling high-quality audio processing directly on the client device. The implementation focuses on optimizing deep learning models for efficient execution on standard hardware, addressing the limitations of existing off-the-shelf solutions and costly server-based approaches.

Datadog developed an internal system to reliably and quickly distribute per-tenant configuration data (context data) to thousands of workload containers. This system addresses challenges of scale, low-latency updates, and resilience, evolving from a Kafka-based invalidation approach to a more robust solution that handles potential failures and high load. The system leverages Consul for configuration distribution and service discovery, with recommendations for optimizing Consul server performance, utilizing ACLs for security, managing watches to prevent self-DDoS, and employing dnsmasq for load lightening.

Datadog developed a feature to automatically detect faulty deployments. The journey involved moving from unlabeled data to supervised learning using weak supervision. Initially, an iterative framework with statistical checks was used to identify deployments with increased error rates, considering impact, temporal correlation, and persistence. To improve time to detection, a sequence of models was introduced, running at different intervals (10, 20, and 60 minutes) after a deployment, each tuned for precision and recall at different stages of data accumulation. The system addresses challenges like lack of labels, data imbalance, and the diversity of application profiles.

This post details the investigation and resolution of a critical segmentation fault in Datadog's Postgres clusters, which was traced back to a bug in the Arm64 JIT compiler (specifically within LLVM). The investigation involved isolating the issue, reproducing it on Arm64 instances, and debugging a Postgres build with JIT debugging support enabled. The root cause was identified as an infinite loop in the Arm64 JIT compilation process for certain expressions, leading to a segfault or a 100% CPU utilization stall. The team successfully identified the problematic assembly code and contributed an upstream fix to LLVM.

Datadog developed a Ruby library for test impact analysis to significantly reduce testing time. Initially, existing solutions like Ruby's Coverage module and TracePoint were explored but found to have high performance overhead. The team then built a custom solution by diving into the Ruby VM and using interpreter events (RUBY_EVENT_LINE) via C extensions. This custom approach achieved a 60-80% performance overhead, compatible with other coverage tools, and offered flexibility for future enhancements. The library aims to seamlessly collect impacted source code files during test runs without requiring code changes from users, ensuring correctness and performance for CI pipelines.

Datadog's static analyzer product, initially built in Java using ANTLR for parsing, has undergone a significant migration to Rust. This migration was driven by performance limitations and incomplete language support with the previous Java-based approach. The new architecture leverages Tree-sitter (implemented in Rust) for AST generation and Deno for JavaScript rule execution, resulting in a threefold performance increase and a tenfold reduction in memory usage. This enables faster scans in resource-constrained CI environments and broader language support.

Datadog's .NET profiler has been enhanced to include memory usage profiling, allowing users to identify high CPU consumption due to excessive garbage collection, pinpoint code responsible for memory allocations, and detect potential memory leaks by tracking surviving objects. The system monitors garbage collector activity by analyzing CLR events and CPU consumption of GC threads. For allocations, it leverages the `AllocationTick` event to sample allocations and capture call stacks. This post details the profiling of exceptions and lock contention, including sampling strategies and CLR event handling.

Datadog developed a heatmap visualization system to effectively represent high-resolution distribution metrics over time at arbitrary scale. This system leverages DDSketch for aggregating data and a frontend strategy of sending bins of counts to achieve constant-time rendering. The visualization allows users to identify distinct 'modes' or patterns within data distributions that might be obscured by traditional percentile graphs, enabling deeper analysis of system behavior and performance. This post details the design and implementation of DDSketch, a novel quantile sketch algorithm that provides relative-error guarantees and is fully mergeable, addressing the limitations of existing algorithms for large-scale, distributed monitoring data.

Datadog developed and continues to evolve DogGraphs, an internal iOS graphing library built with Swift and SwiftUI, to bring complex data visualizations to the Datadog mobile application and widgets. The library focuses on ease of use, a flexible API, default Datadog styling, and high-performance rendering, supporting various widget types and data sources. Recent efforts have concentrated on optimizing SwiftUI rendering pipelines, identifying and fixing performance bottlenecks through tools like _printChanges() and Xcode Instruments, and ensuring compatibility with older iOS versions.

Datadog migrated its internal acceptance tests from a custom Puppeteer-based runner to its own Synthetic Monitoring product. This involved developing a CLI tool (`datadog-ci`) to trigger and manage synthetic tests from CI environments, aiming to reduce flakiness, improve maintainability, and decrease test execution time. The migration process involved identifying pain points through developer surveys and implementing a solution that leverages the existing Synthetic Monitoring platform for recording and executing user interactions.

This post details the root cause analysis of a major platform-level incident on March 8, 2023, which affected all Datadog services across multiple regions. The incident was triggered by a systemd security patch that restarted systemd-networkd. This restart, on Ubuntu 22.04 hosts (which use a newer systemd version), caused systemd-networkd to flush IP rules it did not manage. This behavior, combined with Datadog's Kubernetes networking configuration (where pods receive IPs from the underlying network), led to a global outage. The post also discusses Datadog's incident response process, highlighting its strengths and weaknesses during this event, and emphasizes the importance of a blameless culture and continuous improvement.

Datadog's design system, DRUIDS, has been developed to ensure a consistent, dependable, and repeatable user experience across its expanding platform. It focuses on making components easy to understand, implement, and contribute to, with features like Cmd+K search, DRUIDS Loupe for component inspection, direct links to source code and design tools, editable playgrounds, and a code sandbox for experimentation. The system treats code as the source of truth for component behavior and appearance.

This post details a deep dive into debugging a complex incident involving gRPC, Kubernetes, and AWS networking. It highlights how issues that initially appear to be DNS-related can stem from deeper networking problems, including connection tracking, reverse path filtering, and the interaction between different networking layers (Cilium, Kubernetes, AWS). The post emphasizes the importance of comprehensive visibility through tools like ENA metrics and VPC Flow Logs for diagnosing such issues.

This post details the exploitation of the Dirty Pipe vulnerability to achieve container escape from unprivileged containers, specifically in Kubernetes environments. It explains how the vulnerability allows overwriting files, including the runC binary, and presents a proof-of-concept exploit. The post also discusses defense-in-depth strategies to mitigate such risks.

Datadog contributed to kube-state-metrics to improve its scalability and extensibility. This involved refactoring the metric generation process to be more efficient, reducing collection duration by 15x and enabling more granular data collection at high scale. The contribution focused on improving the core KSM library to handle large Kubernetes clusters more effectively.

Datadog developed an internal tool called 'Clarity' to automate the auditing of third-party SaaS accounts against its HRIS (Workday). This system flags accounts that do not match active employee records, ensuring security and cost-efficiency. It leverages AWS Lambda, Slack, Freshservice, and Datadog for logging, alerting, and ticketing. The tool generates metrics and logs for each flagged account, providing detailed information for investigation and remediation. This post details the initial work to improve cloud security visibility with ChatOps, integrating with Slack, Duo, and PagerDuty to monitor AWS API activity. The pipeline uses Cloudwatch Event Rules, SNS, SQS, and Lambda for cross-account data collection and processing, with Komand used for security orchestration and automation to parse API calls, apply logic, and trigger alerts or notifications. User notifications are sent via Slack with Duo for verification. All workflow details are logged to Elasticsearch for visualization and analysis.

Datadog migrated its job system to Kubernetes, initially experiencing a significant performance regression with jobs completing at a 40-50% slower rate and higher CPU utilization. Through detailed performance experimentation, including careful metric selection (focusing on idle CPU and throughput over load average), the team identified and addressed overheads. Key optimizations involved tuning Kubernetes resource requests (CPU and memory) to improve pod scheduling density, aiming for six pods per node. Analysis of per-pod overhead, including `containerd-shim` CPU and memory usage, revealed it to be relatively small (10ms/pod/second CPU, 1-5MB memory). Further investigation into work execution patterns, such as analyzing `mpstat` output, helped pinpoint CPU utilization issues. The post details the iterative process of experimentation, measurement, and tuning to minimize Kubernetes overhead and restore performance parity with the previous VM-based system.

Datadog, in collaboration with the PHP internals community, contributed to the development of the new observer API in PHP 8. This API addresses limitations of previous observability hooks like zend_execute_ex and custom opcode handlers, such as stack limitations, performance overhead, and incompatibility with the JIT compiler. The new API provides a more robust and performant way for tracers, profilers, and debuggers to instrument PHP applications, enabling better observability without negatively impacting runtime performance.

Datadog has developed Glommio, a Rust crate designed to simplify the implementation of thread-per-core architectures. This model aims to improve application performance and reduce tail latencies by dedicating a thread to each CPU core, thereby eliminating traditional thread context switches and the need for locks when accessing shared data. Glommio addresses the challenges of this paradigm, making it more manageable for developers to build high-throughput systems, particularly for data-intensive applications like datastores.

Datadog's engineering team has explored the application of its monitoring and data analysis capabilities to personal projects, specifically focusing on aggregating and analyzing data from a sailboat. This involved integrating various marine instruments (GPS, autopilot, chartplotter, depth sounder, speedometer, wind transducer, AIS transponder) using NMEA 2000 and SeatalkNG protocols. The goal was to collect comprehensive data for enhanced sailing performance and safety. The project highlights the potential for using Datadog's platform to monitor diverse IoT devices and systems, enabling detailed analysis of operational parameters and environmental conditions.

Datadog has implemented a robust system for the secure publication of Datadog Agent integrations, leveraging TUF and in-toto to ensure end-to-end verification and compromise resilience. This system allows for independent and on-demand release of integrations, enhancing security and agility in the software supply chain.

Datadog is evolving its alerting UX by introducing AI-driven capabilities such as forecasting, anomaly detection, and outlier detection. These methods aim to move beyond static thresholds by adapting to changing conditions, predicting future metric states, and identifying deviations from normal behavior. Algorithmic feeds are also being explored as a way to surface anomalous and outlier behaviors without explicit user configuration, representing a significant shift from opt-in alerting.

Datadog developed Kafka-Kit, a suite of tools to manage and scale Kafka clusters. Kafka-Kit includes `topicmappr` for deterministic partition reassignments, broker replacements, rack-aware placement, and replication factor updates, supporting both 'count' and 'storage' placement strategies. It also includes `autothrottle` for replication auto-throttling. The tools aim to simplify operations for large-scale Kafka deployments, addressing challenges like data movement, capacity planning, and uneven storage utilization.

This post details the use of Datadog APM to identify and resolve performance bottlenecks in the Homebrew package manager. The investigation focused on the `brew linkage` command, which was found to be slow due to the `LinkageChecker::check_dylibs` function. Two optimization solutions were explored: multi-threading (inviable due to the GIL) and caching. An initial caching mechanism using SQLite3 was implemented, significantly reducing execution time from 11.5 seconds to 182 milliseconds for 106 packages. A subsequent implementation using Ruby's `pstore` was also explored as a dependency-free alternative.

This post details the initial integration of CPython into the Datadog Agent, a Go binary. It explains how cgo is used to bridge Go and C APIs, enabling the embedding of the Python interpreter. The article covers initializing and finalizing the interpreter, importing modules, and calling Python functions from Go, highlighting the use of the go-python library as a wrapper for cgo details. It also touches upon the Global Interpreter Lock (GIL) as a consideration for concurrency.

Datadog has developed a robust system for managing multiple AWS accounts securely and efficiently. This system leverages IAM users, groups, and roles to enforce granular access control, requiring MFA for privileged operations and grouping permissions by topic. The approach aims to balance security with usability, allowing for effective management of infrastructure across numerous accounts.

This post introduces the concept and application of robust statistical distances for machine learning, particularly for outlier and anomaly detection. It details various distance metrics like Kolmogorov-Smirnov, Earth Mover's, and Cramér-von Mises, and provides interactive visualizations to compare different probability distributions. The implementation uses D3.js for visualization and demonstrates how these distances can be used to build more powerful ML algorithms.

Datadog's Solutions Team developed a system for reproducing customer environments using Vagrant for local VMs and Terraform for remote cloud instances (AWS). This system standardizes environment setup through shared provisioning scripts (`setup.sh`) and data files, enabling faster issue reproduction, team collaboration, and efficient demonstrations. Future plans include further modularization of setup scripts and leveraging Terraform backends like Consul for state management.

This post details the implementation of piecewise linear regression, a technique used to model data that exhibits different linear trends across different ranges. The implementation involves visualizing raw data, demonstrating the evolution of regression segments, and calculating the cost associated with different segmentation points. The interactive visualization allows users to explore how the model adapts to the data and identify optimal segmentation.

This post details the investigation into the performance of Protocol Buffers (protobuf) serialization in Python, particularly in the context of extracting metrics from kube-state-metrics. It covers the basics of protobuf, including defining messages, generating code, and handling streaming multiple messages by prepending message size. The post benchmarks payload sizes for protobuf versus plain text and highlights the performance limitations of the pure Python protobuf implementation, suggesting the use of C++ extensions for significant speed improvements.

The Datadog Agent has evolved to handle potential system hangs caused by `os.statvfs` calls on NFS mounts. Initially, the agent would stall during disk checks when encountering problematic NFS mounts, leading to gaps in metrics. The system was improved by running `os.statvfs` on a separate thread and implementing a timeout mechanism to prevent the main agent thread from hanging. This ensures continued operation even with unreliable NFS connections, though it may slightly increase memory usage on affected systems.

Datadog developed Redux-Doghouse, a library to create reusable React-Redux components by scoping actions and reducers. This allows multiple instances of a component to operate independently within a larger Redux application, preventing action conflicts. The library was used to rebuild Datadog's Query Editor and Expression Editor, enabling reuse across different parts of the application while maintaining independent functionality.

Datadog engineers developed a system to monitor bathroom occupancy using IoT devices. The system utilizes Raspberry Pis connected to magnetic reed switches and pin switches to detect door lock status. Data is exposed via a Python script running under tcpserver and daemontools, accessible via Netcat. This allows for real-time status updates to help reduce bathroom contention in the office.

Datadog has developed and released Go bindings for the czlib (zlib wrapper) and zstd compression libraries. These bindings are designed to improve performance for data pipelines that rely on compression, offering faster compression and decompression rates compared to pure Go implementations. The czlib binding is a fork of vitess's cgzip, adapted for zlib wrapping. The zstd binding mimics the zlib interface and exposes advanced features like stream compression and compression levels. Benchmarks demonstrate significant performance gains, especially for larger messages.