Datadog Feature Trails

Feature trailActive

Distributed DataFusion

Datadog has developed Distributed DataFusion, an open-source framework that extends Apache DataFusion to execute a single query across multiple machines. This addresses the limitation of DataFusion executing queries on a single node, enabling Datadog to handle large interactive workloads by distributing compute resources. The system aims for low-latency interactive execution with zero-copy streaming and efficient use of resources at scale, integrating with Apache Arrow and Substrait. Benchmarks show significant performance improvements over existing distributed engines like Spark and Trino for heavy queries, while lightweight queries remain on a single machine to avoid overhead. The framework extends DataFusion's physical execution model across workers, building on its partitioning capabilities to coalesce results from multiple machines.

1 post · 2026

Timeline

Feature trailActive

Java CPU Profiling

Datadog's APM profilers have evolved to provide unbiased CPU and async task profiling. For Java, this involved collaborating with OpenJDK to introduce a new CPU profiling event. For Python, a new async-aware profiling model reconstructs relationships between asyncio tasks, addressing limitations of traditional flame graphs and reducing profiler overhead. This aims to provide more accurate and actionable performance insights for complex asynchronous applications.

8 posts · 2020–2026

Timeline

20202026

Feature trailActive

Rust Observability with OpenTelemetry

Datadog is evolving its observability stack for Rust services by contributing to OpenTelemetry and building dd-trace-rs, an opinionated Rust tracer. This aims to provide consistent tracing, context propagation, and sampling across services, improving incident investigation and reducing resource consumption. Key efforts include improving interoperability between the `tracing` crate and OpenTelemetry, and extending the OpenTelemetry pipeline with opinionated configurations for Datadog users. The new dd-trace-rs library builds on the OpenTelemetry Rust SDK, offering opinionated configurations for Datadog users, including automatic configuration via environment variables, consistent sampling strategies, and reliable context propagation. Significant upstream contributions to OpenTelemetry Rust and `tracing-opentelemetry` have improved interoperability between the `tracing` and OpenTelemetry APIs, fixing issues with context synchronization and span stacking, benefiting the broader Rust ecosystem.

1 post · 2026

Timeline

Feature trailActive

Git Serving Infrastructure

Datadog rebuilt its Git serving infrastructure with 'gitretriever' to handle a 20x increase in CI traffic. The new system uses independent pods with 'mirrors' to sync with GitHub and 'relays' to serve code to CI workloads. This architecture avoids the replication overhead of previous systems and reduces CPU load by distributing Git pulls across branches and optimizing packfile generation. Key principles include parallel fetching for changed references, serving local copies directly without conse

1 post · 2026

Timeline

Feature trailActive

Java APM Class Matching Optimization

Datadog's APM Java agent has significantly improved its class-matching performance during startup. This involved optimizing the process of identifying which classes to instrument for observability. The team developed a novel approach by encoding a prefix trie, used for matching class name prefixes, directly as a JVM constant. This allows the trie to be loaded with a single bytecode instruction, eliminating the need for resource lookups and file parsing during the constrained `premain` phase.

1 post · 2026

Timeline

Feature trailActive

Data Completeness Measurement

Datadog built a system to measure data completeness across all ingestion pipelines in real-time for every customer. The system tracks payloads end-to-end by splitting pipelines into segments and counting creates and acknowledgments for unique payload identifiers within time buckets. This allows for localization of degradation and aggregation into an end-to-end view, handling complex pipeline structures and edge cases like delayed data and retries. The Stream Router system, responsible for data i

9 posts · 2018–2026

Timeline

20182026

Feature trailActive

Multi-tenant Data Replication Platform

Datadog built a managed data replication platform to deliver highly reliable, highly scalable, and flexible data movement across the company. This platform abstracts operational overhead, provides robust monitoring and alerting, and adapts to new use cases. It evolved from addressing performance issues in a shared Postgres database by rerouting search queries to a dedicated search platform with dynamically denormalized data. The provisioning process was automated using Temporal workflows, breaking down monolithic data models into smaller, more manageable relational schemas. This post details a migration of the Stream Router's storage backend from a KV model (FoundationDB/RocksDB) to a relational model (PostgreSQL for writes, DuckDB for reads), leveraging AI-assisted refactoring to accelerate the process and ensure live traffic continuity. The new architecture supports complex relational queries and avoids transaction size limits.

4 posts · 2025–2026

Timeline

20252026

Feature trailActive

Postgres Upsert Performance Optimization

Datadog's PostgreSQL high-availability architecture on Kubernetes has been re-architected for failover safety. The system now uses synchronous replication for failover candidates, coordinated by Patroni, to ensure durability and prevent data loss during network partitions or zonal failures. This involves reconfiguring the leader pool to prioritize data consistency over immediate availability when network latency spikes, ensuring that replicas are sufficiently up-to-date before a failover can occur. This post details the debugging of an upsert query that unexpectedly doubled disk writes and quadrupled WAL syncs by writing to WAL even when no data changed, due to row locking. The solution involved creating a dedicated table for `last_ingested` timestamps and optimizing the upsert query to leverage Heap-Only Tuples (HOT) updates by ensuring updates only modify unindexed columns and by setting an appropriate fillfactor, thereby reducing write amplification and WAL activity.

1 post · 2025–2026

Timeline

20252026

Feature trailActive

Malicious Code Detection

Datadog's BewAIre system has evolved from detecting malicious code in pull requests to scanning entire dependency packages and upstream package registries. This expansion leverages a two-stage evaluation pipeline: a fast, inexpensive filter phase using a previous-generation or high-speed LLM, followed by a more thorough agentic investigation phase for flagged changes. The investigation phase utilizes tools to gather additional context, such as GitHub API calls for commit history, contributor analysis, and code execution sandboxing. The system is now integrated into the CI/CD pipeline to review all pull requests in real-time, classifying them as benign or malicious with a low false positive rate. This evolution aims to address the growing attack surface and reviewer fatigue associated with AI-assisted development.

3 posts · 2025–2026

Timeline

20252026

Feature trailActive

Screenshot Watermarking

Datadog introduced a steganography-based system to embed widget metadata within screenshots. This system allows for the invisible encoding of a Redis cache key into the pixel data of a widget's border. When a screenshot is taken and pasted, the embedded key can be used to retrieve the original widget's metadata from Redis, enabling features like live previews and context preservation even when users opt for screenshots over direct sharing links. The implementation involves compressing and encoding metadata into an 8-byte key, which is then embedded into the 1px border of a widget. This key is used to retrieve the full widget definition from a Redis cache. The system handles over a billion widget renders per day and ensures collision-free watermarking by prepending organization IDs to widget IDs. Performance optimizations allow for rendering over a billion watermarks per day without impacting app responsiveness.

1 post · 2026

Timeline

Feature trailActive

Autonomous SRE Agent Evaluation Platform

Datadog is evolving its autonomous SRE agent capabilities by building real-world evaluation platforms for agents like Bits Investigation. This platform enables reproducible investigation environments, using a curated label set of representative investigations and an orchestration platform to score agent performance. The goal is to address subtle regressions in agent behavior and scale the creation of evaluation labels by embedding it into the product. The platform now incorporates a pipeline for generating LLM-based postmortem drafts, integrating structured incident metadata with unstructured Slack discussions to produce accurate and reliable drafts while maintaining human author control. This involves an experimentation framework for data extraction and LLM instruction refinement, alongside LLM-generated incident summaries for rapid onboarding.

1 post · 2024–2026

Timeline

20242026

Feature trailActive

AI Agent Observability Interface (MCP)

Datadog's MCP (Model Context Protocol) server has evolved significantly to provide an observability interface specifically designed for AI agents. Initial versions were simple API wrappers, but the system has been re-architected to address agent constraints like context window efficiency, data retrieval limitations, and tool management. Key developments include optimizing data formats (CSV, YAML over JSON), implementing token-budget-based pagination, introducing query languages (SQL) for data analysis, and improving error messages and documentation discoverability. The platform supports flexible tools, toolsets, and layering for specialized needs, while also exploring the integration of agent skills. The goal is to make observability ergonomic for agents, reducing token usage and improving accuracy.

2 posts · 2024–2026

Timeline

20242026

Feature trailActive

Datadog Agent Binary Size Reduction

2 posts · 2025–2026

Timeline

20252026

Feature trailActive

eBPF-based File Integrity Monitoring

Datadog developed an eBPF-based File Integrity Monitoring (FIM) system to address the limitations of traditional methods like periodic scans, inotify, and auditd. The system ingests billions of file-related kernel events per minute and employs a multi-stage filtering approach. Initially, agent-side rules discard irrelevant events, reducing the volume to approximately one million events per minute. Further optimization involves moving significant event evaluation logic into eBPF programs to pre-filter 94% of events directly in the kernel, reducing the data volume sent to the backend and improving scalability and performance.

2 posts · 2025–2026

Timeline

20252026

Feature trailActive

Graceful Degradation and Reliability Engineering

Datadog is evolving its approach to reliability by prioritizing graceful degradation over preventing all failures. This involves implementing persistent intake storage to prevent data loss, making live data available faster by skipping backlogs and prioritizing critical telemetry, and updating retry logic to avoid overwhelming downstream systems. The company is also addressing architectural bottlenecks and technical debt, and introducing prioritization at the infrastructure and compute level to ensure linear degradation of throughput with available compute capacity, preventing complete service failure during resource constraints. This is exemplified by the design of Courier, a new message queuing service, which incorporates multi-tenancy, at-least-once delivery, graceful degradation, and horizontal scalability, utilizing formal modeling, lightweight simulations, and chaos testing to validate its design.

4 posts · 2023–2025

Timeline

20232026

Feature trailCooling

Husky Query Engine

Datadog's Husky event store has evolved its query engine and underlying data storage layer to handle trillions of events per day. This includes optimizing compaction strategies for efficient merging of small fragments into larger ones, balancing query performance, parallelism, and storage layout. The system uses a columnar storage format similar to Parquet, with custom optimizations for observability data, including a skip list for column offsets to quickly find specific columns. Compaction is performed lazily and efficiently, aiming for a fixed amount of memory and saturating CPU while minimizing storage API requests. The goal is to reduce the number of fragments fetched from object storage and the number of events scanned per query.

6 posts · 2022–2025

Timeline

20222026

Feature trailCooling

Agentic Code Optimization

Datadog is developing BitsEvolve, an internal agentic system for self-optimizing code. This system leverages evolutionary algorithms to mutate code, evaluate variants against benchmarks (fitness function), and iterate. It employs an island model for parallel evolution and sharing of top performers. This builds upon manual code optimization efforts in critical hot-path functions, aiming to scale deep optimization work beyond specialized expertise.

1 post · 2025

Timeline

20252026

Feature trailCooling

Live Process and Container Metrics Real-time Data Pipeline

Datadog's live process and container metrics pipeline has been significantly optimized. Initially, it collected data from all hosts in a tenant's infrastructure whenever a user viewed the Processes or Containers pages, leading to millions of data points per second. The system has evolved to collect real-time data only from hosts actively being viewed by the user (up to 50 hosts), drastically reducing data volume. This was achieved by filtering data based on 'host subscriptions' and propagating t

1 post · 2025

Timeline

20252026

Feature trailCooling

Real-time Timeseries Storage Engine

Datadog has evolved its real-time timeseries storage engine for the sixth generation, building a new system in Rust. This iteration focuses on high throughput and low latency, aiming to address challenges posed by increasing data volume, complexity, and cardinality. The new engine is designed to handle high-cardinality workloads, complex queries, and bursty traffic patterns, achieving significant performance gains in ingestion and query speed. The architecture continues to separate real-time data. This post details the evolution of the timeseries indexing service, comparing two generations of indexing services and their impact on query performance and maintenance. It highlights the challenges of timeseries indexing at scale, including the trade-offs between query performance, reliability, and maintenance overhead. The new indexing strategy focuses on optimizing index selectivity and managing index lifecycle to improve query efficiency and reduce operational burden.

1 post · 2024–2025

Timeline

20242026

Feature trailCooling

Go Map Implementation Evolution

Datadog is continuously optimizing its Go services by tracking and leveraging improvements in the Go runtime. This includes investigating memory regressions and identifying performance gains from new language features, such as the Swiss Tables implementation for maps, leading to significant memory savings in high-traffic environments.

1 post · 2025

Timeline

20252026

Feature trailCooling

Go Map Memory Optimization

Datadog's Go Agent binaries have undergone significant size reductions, decreasing by up to 77% through systematic dependency auditing, targeted refactors, and re-enabling linker optimizations. This effort involved analyzing build tag usage, isolating dependencies into separate packages, and leveraging tools like `go list` and `goda` for dependency graphing. The project also led to contributions to the Go toolchain and improvements for other large Go projects like Kubernetes. This post details how Go 1.24's Swiss Tables implementation reduced memory usage in a large in-memory map, showing how Datadog profiled and sized the change, and sharing struct-level optimizations that led to even larger fleet-wide savings.

1 post · 2025

Timeline

20252026

Feature trailCooling

Client-Side Noise Suppression Library

Datadog developed and open-sourced dtln-rs, a real-time, client-side noise suppression library based on the Dual-Signal Transformation LSTM Network (DTLN). This library is designed to be embeddable in native clients and web applications, offering high performance without server dependencies. It leverages WebAssembly, Node.js native modules, and native Rust targets to integrate with WebRTC, enabling high-quality audio processing directly on the client device. The implementation focuses on optimizing STFT, phase information, and LSTM neural networks for efficient, real-time noise reduction.

1 post · 2025

Timeline

20252026

Feature trailCooling

Configuration Distribution System

Datadog uses Consul for distributing configuration and service discovery across its microservices architecture. Key recommendations include provisioning adequate CPU for Consul servers to minimize leadership transitions, using git2consul for auditable configuration changes, leveraging ACLs for security, employing Consul watches with safeguards like sifter to prevent excessive firing, optimizing DNS lookups with dnsmasq and caching for high-volume scenarios, and implementing comprehensive monitoring. This post details the evolution of a system for fast, reliable configuration distribution to thousands of workload containers, moving from direct database connections with Kafka-based cache invalidation to a more resilient architecture that addresses the challenges of scale and failure modes in a multi-tenant distributed environment.

2 posts · 2016–2025

Timeline

…20172026

Feature trailCooling

Faulty Deployment Detection

Datadog developed a feature to automatically detect faulty deployments. The journey involved moving from unlabeled data to supervised learning using weak supervision. Initially, an iterative framework with statistical checks was used to identify deployments with increased error rates, considering impact, temporal correlation, and persistence. To improve time to detection, a sequence of models was introduced, running at different intervals (10, 20, and 60 minutes) after a deployment, each tuned differently. This post details the evolution from an unsupervised approach to a sequence of supervised models to improve time to detection and recall.

1 post · 2025

Timeline

20252026

Feature trailCooling

Datadog Lambda Extension (Rust)

Datadog's AWS Lambda extension has been re-architected in Rust to significantly improve performance and reduce overhead. This rewrite cut cold starts by 82%, reduced memory usage by 40%, and shrunk the binary size from 55 MB to 7 MB. The new extension operates as a sidecar process, collecting and relaying telemetry data asynchronously. It supports flexible flush strategies (end of invocation, race the invocation, periodically) to balance telemetry immediacy with minimal overhead. The design constraints focused on minimizing impact on the Lambda function handler and post-runtime duration, leading to tradeoffs like manually writing AWS API calls instead of using SDKs.

1 post · 2025

Timeline

20252026

Feature trailCooling

Kafka Scaling Tools

Datadog's Streaming Platform provides a control layer for managing massive, multi-cluster Kafka deployments. It abstracts Kafka complexity with 'Streams' for resilient, multi-cluster pipelines, a custom 'Assigner' for real-time coordination and failovers, and 'Stream lanes' for QoS and dead-letter handling. The platform also introduces an advanced commit log to overcome head-of-line blocking and a custom Kafka client, 'libstreaming', for performance and observability. This enables dynamic workload management and real-time reliability at scale.

1 post · 2018–2025

Timeline

20182026

Feature trailCooling

Postgres JIT Compiler Bug Investigation

Datadog's engineering team investigated and resolved a critical segmentation fault in their Postgres clusters. The issue was traced to a bug in the Arm64 JIT compiler within LLVM. The investigation involved isolating the problem, reproducing it on Arm64 instances, and debugging a Postgres build with JIT debugging support enabled. The root cause was identified as an infinite loop in the Arm64 JIT compilation process for certain expressions, leading to a segfault or a 100% CPU utilization. An upstream fix was developed and applied.

1 post · 2025

Timeline

20252026

Feature trailCooling

Ruby Test Impact Analysis

Datadog developed a Ruby library for test impact analysis to significantly reduce testing time. Initially, existing solutions like Ruby's Coverage module and TracePoint were explored but found to have high performance overhead. The team then built a custom solution by diving into the Ruby VM and using interpreter events (RUBY_EVENT_LINE) via C extensions. This custom approach achieved a 60-80% performance overhead, compatible with other coverage tools, and offered flexibility for future enhancements.

1 post · 2024

Timeline

20242026

Feature trailQuiet since 2024

Static Analyzer Migration to Rust

Datadog's static analyzer product, initially built in Java using ANTLR for parsing, has undergone a significant migration to Rust. This migration was driven by performance limitations and incomplete language support with the previous Java-based approach. The new architecture leverages Tree-sitter (implemented in Rust) for AST generation and Deno for JavaScript rule execution, resulting in a threefold performance increase and a tenfold reduction in memory usage. This enables faster scans in resource-constrained environments like CI systems.

1 post · 2024

Timeline

20242026

Feature trailQuiet since 2024

.NET Memory Profiling

Datadog's .NET profiler has evolved to include memory usage profiling, enabling users to identify high CPU consumption from excessive garbage collection, pinpoint code responsible for memory allocations, and detect potential memory leaks by tracking surviving objects. This involves monitoring GC activity via CLR events, sampling allocations using the `AllocationTick` event, and tracking object lifetimes using weak handles and the `ICorProfilerInfo13` API. The system also addresses upscaling challenges by estimating real values based on sampled allocations and their sizes.

4 posts · 2024

Timeline

20242026

Feature trailQuiet since 2024

Heatmap Visualization for Distributions

Datadog developed a heatmap visualization system to effectively represent high-resolution distribution metrics over time at arbitrary scale. This system leverages DDSketch for aggregating data and a frontend strategy of sending bins of counts to achieve constant-time rendering. The visualization allows users to identify distinct 'modes' or patterns within data distributions that might be obscured by traditional percentile graphs, enabling deeper analysis of system behavior and performance.

2 posts · 2019–2024

Timeline

20192026

Feature trailQuiet since 2024

iOS Data Visualization Library

Datadog developed and continues to evolve DogGraphs, an internal iOS graphing library built with Swift and SwiftUI, to bring complex data visualizations to the Datadog mobile application and widgets. The library focuses on ease of use, a flexible API, default Datadog styling, and high-performance rendering, supporting various widget types and data sources. Recent efforts have concentrated on optimizing SwiftUI rendering pipelines, identifying and fixing performance bottlenecks through tools like Xcode Instruments, and leveraging private APIs like _printChanges() for debugging.

1 post · 2024

Timeline

20242026

Feature trailQuiet since 2023

Synthetic Monitoring for Acceptance Tests

Datadog migrated its internal acceptance tests from a custom Puppeteer-based runner to its own Synthetic Monitoring product. This involved developing a CLI tool (`datadog-ci`) to trigger and manage synthetic tests from CI environments, aiming to reduce flakiness, improve maintainability, and decrease test execution time. The migration process involved identifying pain points through developer surveys and implementing a solution that leverages the existing Synthetic Monitoring platform for recording and executing tests.

1 post · 2023

Timeline

20232026

Feature trailQuiet since 2023

Systemd Network Configuration and Outage Root Cause Analysis

Datadog experienced a major platform-level incident on March 8, 2023, affecting all services across multiple regions. The incident was triggered by a systemd security patch that restarted systemd-networkd. This restart caused systemd-networkd to flush IP rules it did not manage, leading to network connectivity loss for nodes running Kubernetes control planes and application workloads. The recovery process involved rebooting parent control plane nodes, then child control plane pods, and finally a coordinated reboot of all affected nodes.

2 posts · 2023

Timeline

20232026

Feature trailQuiet since 2022

Datadog Design System (DRUIDS)

Datadog's design system, DRUIDS, has been developed to ensure a consistent, dependable, and repeatable user experience across its expanding platform. It focuses on making components easy to understand, implement, and contribute to, with features like Cmd+K search, DRUIDS Loupe for component inspection, direct links to source code and design tools, editable playgrounds, and a code sandbox for experimentation. The system treats code as the source of truth for component behavior and appearance.

1 post · 2022

Timeline

20222026

Feature trailQuiet since 2022

SaaS Account Auditing and Monitoring

Datadog developed an internal tool called 'Clarity' to automate the auditing of third-party SaaS accounts against its HRIS (Workday). This system flags accounts that do not match active employee records, ensuring security and cost-efficiency. The tool generates metrics and logs for each flagged account, providing detailed information for investigation and remediation. The system was expanded into Clarity License Manager (CLM) to optimize software licensing efficiency, monitor account activity, and provide a centralized dashboard for IT teams.

2 posts · 2017–2022

Timeline

20172026

Feature trailQuiet since 2022

Kubernetes and AWS Networking Debugging

Datadog has developed a robust system for managing multiple AWS accounts securely and efficiently. This system leverages IAM users, groups, and roles to enforce granular access control, requiring MFA for privileged operations and grouping permissions by topic. The approach aims to balance security with usability, allowing for effective management of infrastructure across numerous accounts. This post details a deep dive into debugging a complex incident involving gRPC, Kubernetes, and AWS networking. It highlights how issues that initially appear to be DNS-related can stem from deeper networking problems, including connection tracking, reverse path filtering, and the interaction between different networking layers (Cilium, Kubernetes, AWS). The post emphasizes the importance of comprehensive visibility through tools like ENA metrics and VPC Flow Logs for diagnosing such issues.

1 post · 2022

Timeline

20222026

Feature trailQuiet since 2022

Container Escape Vulnerability Research

Datadog Security Labs researches and demonstrates container escape vulnerabilities, specifically focusing on how new kernel flaws like Dirty Pipe can be exploited to compromise host systems from within unprivileged containers. This includes analyzing the attack vectors, developing proof-of-concept exploits, and outlining defense-in-depth strategies to mitigate such risks.

1 post · 2022

Timeline

20222026

Feature trailQuiet since 2022

Go Profiling Improvements

Datadog is enhancing its Go profiling capabilities by contributing to and leveraging improvements in the Go runtime. This includes addressing CPU profiling accuracy issues on Linux by migrating from setitimer(2) to timer_create(2) for more reliable signal delivery and per-thread accounting. Additionally, bug fixes have been implemented for profiler labels (pprof tags), improving their reliability and enabling more granular data slicing for tracing and debugging. These efforts aim to provide more accurate and actionable performance insights for Go applications.

1 post · 2022

Timeline

20222026

Feature trailQuiet since 2022

User Experience Visibility with RUM

Datadog is enhancing its ability to gain visibility into the user experience of its own platform by leveraging its Real User Monitoring (RUM) product, logs, and notebooks. This involves collecting quantitative data on user interactions to inform design decisions, such as optimizing font choices and refining UI component functionality. The goal is to improve the overall user experience by understanding how users interact with the platform and identifying areas for improvement.

1 post · 2022

Timeline

20222026

Feature trailQuiet since 2021

Kubernetes State Metrics Collection

Datadog contributed to kube-state-metrics to improve its scalability and extensibility. This involved refactoring the metric generation process to be more efficient, reducing collection duration by 15x and enabling more granular data collection at high scale. The contribution focused on improving the core KSM library to handle large Kubernetes clusters more effectively.

1 post · 2021

Timeline

20212026

Feature trailQuiet since 2021

Akka ForkJoinPool Optimization

Datadog is optimizing its Akka applications by analyzing and addressing performance bottlenecks related to the ForkJoinPool. This involves understanding how the pool dynamically manages threads and identifying scenarios where irregular task flow combined with high thread counts can lead to excessive CPU usage due to frequent thread parking and unparking. Solutions include reconfiguring dispatchers and limiting actor instances to stabilize load and reduce overhead.

1 post · 2021

Timeline

20212026

Feature trailQuiet since 2021

Kubernetes Job System Performance Optimization

Datadog migrated its job system to Kubernetes, initially experiencing a significant performance regression with jobs completing at a 40-50% slower rate and higher CPU utilization. Through detailed performance experimentation, including careful metric selection (focusing on idle CPU and throughput over load average), the team identified and addressed overheads. Key optimizations involved tuning Kubernetes resource requests (CPU and memory) to improve pod scheduling density, aiming for six pods per node.

1 post · 2021

Timeline

20212026

Feature trailQuiet since 2021

PHP Observability API

Datadog, in collaboration with the PHP internals community, contributed to the development of the new observer API in PHP 8. This API addresses limitations of previous observability hooks like zend_execute_ex and custom opcode handlers, such as stack limitations, performance overhead, and incompatibility with the JIT compiler. The new API provides a more robust and performant way for tracers, profilers, and debuggers to instrument PHP applications, enabling better observability without negatively impacting performance.

1 post · 2021

Timeline

20212026

Feature trailQuiet since 2020

Thread-per-core programming model

Datadog has developed Glommio, a Rust crate designed to simplify the implementation of thread-per-core architectures. This model aims to improve application performance and reduce tail latencies by dedicating a thread to each CPU core, thereby eliminating traditional thread context switches and the need for locks when accessing shared data. Glommio addresses the challenges of this paradigm, making it more manageable for developers to build high-throughput systems, particularly for data-intensive applications.

1 post · 2020

Timeline

20202026

Feature trailQuiet since 2020

Marine IoT Data Aggregation and Monitoring

Datadog's engineering team has explored the application of its monitoring and data analysis capabilities to personal projects, specifically focusing on aggregating and analyzing data from a sailboat. This involved integrating various marine instruments (GPS, autopilot, chartplotter, depth sounder, speedometer, wind transducer, AIS transponder) using NMEA 2000 and SeatalkNG protocols. The goal was to collect comprehensive data for enhanced sailing performance and safety. The project highlights the use of Datadog's platform for real-world IoT data aggregation and monitoring.

1 post · 2020

Timeline

20202026

Feature trailQuiet since 2019

Secure Software Publication

Datadog has implemented a robust system for the secure publication of Datadog Agent integrations, leveraging TUF and in-toto to ensure end-to-end verification and compromise resilience. This system allows for independent and on-demand release of integrations, enhancing security and agility in the software supply chain. This post details the integration of TUF for secure transport and key management, and in-toto for end-to-end supply chain verification, with developers signing integrations using Yubikeys and GPG.

1 post · 2019

Timeline

20192026

Feature trailQuiet since 2019

AI-driven Alerting UX

Datadog is evolving its alerting UX by introducing AI-driven capabilities such as forecasting, anomaly detection, and outlier detection. These methods aim to move beyond static thresholds by adapting to changing conditions, predicting future metric states, and identifying deviations from normal behavior. Algorithmic feeds are also being explored as a way to surface anomalous and outlier behaviors without explicit user configuration, representing a significant shift from opt-in alerting.

1 post · 2019

Timeline

20192026

Feature trailQuiet since 2018

Email Event Logging and Monitoring

Datadog has implemented a system to ingest and monitor email events from Amazon SES, enabling enhanced observability for critical communications like password resets. This system leverages AWS SNS and Lambda to forward SES events to Datadog Log Management, allowing for easy searching, faceting, and alerting on email delivery status. This provides the support and security teams with direct access to email-related metrics, improving troubleshooting and trust in the email sending infrastructure.

1 post · 2018

Timeline

20182026

Feature trailQuiet since 2018

Homebrew Performance Optimization

Datadog's APM has been used to identify and resolve performance bottlenecks in the Homebrew package manager. The investigation focused on the `brew linkage` command, which was found to be slow due to the `LinkageChecker::check_dylibs` function. Two optimization solutions were explored: multi-threading (inviable due to the GIL) and caching. An initial caching mechanism using SQLite3 was implemented, significantly reducing execution time from 11.5 seconds to 182 milliseconds for 106 packages. A subsequent implementation using Ruby's `pstore` was also explored as an alternative without third-party gem dependencies.

1 post · 2018

Timeline

20182026

Feature trailQuiet since 2018

Embedding Python in Go

Datadog's Go Agent embeds a CPython interpreter to execute Python checks and custom scripts without requiring recompilation. This integration leverages cgo to bridge Go and C APIs, allowing for programmatic operations like initializing and finalizing the interpreter, importing modules, and calling Python functions. The use of the go-python library abstracts cgo details for a more idiomatic Go experience. The article also discusses the Global Interpreter Lock (GIL) as a consideration for concurrency when embedding Python.

1 post · 2018

Timeline

20182026

Feature trailQuiet since 2017

Multi-account IAM and Security

1 post · 2017

Timeline

20172026

Feature trailQuiet since 2017

Statistical Distances for ML

This post introduces the concept and application of robust statistical distances for machine learning, particularly for outlier and anomaly detection. It details various distance metrics like Kolmogorov-Smirnov, Earth Mover's, and Cramér-von Mises, and provides interactive visualizations to compare different probability distributions. The implementation uses D3.js for visualization and demonstrates how these distances can be used to build more powerful ML algorithms.

1 post · 2017

Timeline

20172026

Feature trailQuiet since 2017

Environment Reproduction and Management

Datadog's Solutions Team developed a system for reproducing customer environments using Vagrant for local VMs and Terraform for remote cloud instances (AWS). This system standardizes environment setup through shared provisioning scripts (`setup.sh`) and data files, enabling faster issue reproduction, team collaboration, and efficient demonstrations. Future plans include further modularization of setup scripts and leveraging Terraform backends like Consul for state management.

1 post · 2017

Timeline

20172026

Feature trailQuiet since 2017

Cloud Security Monitoring and Alerting Pipeline

Datadog has built a serverless security monitoring and alerting pipeline for its AWS cloud operations. This pipeline integrates with Slack, Duo, and PagerDuty to notify, alert, and authenticate potential security-relevant API calls. It centralizes data from multiple AWS accounts into a dedicated security account, using CloudWatch Event Rules, SNS, and SQS for cross-account data collection. A Lambda function triggers every two minutes to process API call data, which is then fed into a security orchestration framework (Komand) for logic processing, decision-making, and user notifications. The system categorizes API calls by severity (log, notify, alert) and uses interactive Slack messages with Duo authentication for user verification. All workflow details are logged to Elasticsearch for visualization and analysis.

1 post · 2017

Timeline

20172026

Feature trailQuiet since 2017

Piecewise Regression Implementation

Datadog developed a system for implementing piecewise linear regression, a technique used to model data that exhibits different linear trends across different ranges. The implementation involves visualizing raw data, demonstrating the evolution of regression segments, and calculating the cost associated with different segmentation points. The interactive visualization allows users to explore how the model adapts to the data and identify optimal segmentation.

1 post · 2017

Timeline

20172026

Feature trailQuiet since 2017

Protobuf Serialization Performance

Datadog has implemented and continues to optimize the use of Protocol Buffers (protobuf) for data serialization, particularly within the Datadog Agent for extracting metrics from systems like kube-state-metrics. This involves understanding protobuf's serialization mechanisms, including handling streaming messages by prepending message size, and benchmarking its performance against other formats like plain text and Prometheus text format. The team has investigated the performance of the pure Python protobuf implementation and the benefits of using C++ extensions for significant performance gains. This work contributes to efficient data handling and reduced payload sizes for observability data.

1 post · 2017

Timeline

20172026

Feature trailQuiet since 2017

Datadog Agent Mount Handling

The Datadog Agent has evolved to handle potential system hangs caused by `os.statvfs` calls on NFS mounts. Initially, the agent would stall during disk checks when encountering problematic NFS mounts, leading to gaps in metrics. The system was improved by running `os.statvfs` on a separate thread and implementing a timeout mechanism to prevent the main agent thread from hanging. This ensures continued operation even with unreliable NFS connections, though it may slightly increase memory usage on systems that have a hard mounted NFS disks.

1 post · 2017

Timeline

20172026

Feature trailQuiet since 2016

Reusable React-Redux Component Scoping

Datadog developed Redux-Doghouse, a library to create reusable React-Redux components by scoping actions and reducers. This allows multiple instances of a component to operate independently within a larger Redux application, preventing action conflicts. The library was used to rebuild Datadog's Query Editor and Expression Editor, enabling reuse across different parts of the application while maintaining independent functionality.

1 post · 2016

Timeline

…20172026

Feature trailQuiet since 2016

IoT Bathroom Occupancy Monitoring

Datadog engineers developed a system to monitor bathroom occupancy using IoT devices. The system utilizes Raspberry Pis connected to magnetic reed switches and pin switches to detect door lock status. Data is exposed via a Python script running under tcpserver and daemontools, accessible via Netcat. This allows for real-time status updates to help reduce bathroom contention in the office.

1 post · 2016

Timeline

…20172026

Feature trailQuiet since 2016

Go Compression Libraries (czlib and zstd)

Datadog has developed and released Go bindings for the czlib (zlib wrapper) and zstd compression libraries. These bindings are designed to improve performance for data pipelines that rely on compression, offering faster compression and decompression rates compared to pure Go implementations. The czlib binding is a fork of vitess's cgzip, adapted for zlib wrapping. The zstd binding mimics the zlib interface and exposes advanced features like stream compression and compression levels. Benchmarks show significant performance gains.

1 post · 2016

Timeline

…20172026