Observability Platform

8/4/2026

This post introduces the Cloudflare Codex, a governed set of engineering standards designed to be consumed by both humans and AI agents. It details the Codex's organization into domains, its RFC-based format with SHOULD/MUST keywords, and its lifecycle states (approved/enforced). The post explains how the Codex supports engineering workflows through consumers like the AI code reviewer (which has flagged 230,000+ violations), the spec reviewer (which has evaluated ~600 technical designs), and the incident report reviewer (which assesses postmortems). It also highlights the development of language-specific linters and a local CLI for the AI code reviewer to improve developer experience. The Codex is built on the Developer Platform, utilizing Workers, D1, and AI Gateway.

8/4/2026

Introduces Cloudflare Agents, a platform for deploying and managing hosted agents, starting with observability. Key features include agent tracing for detailed visibility into agent behavior (model calls, tool execution, token usage), integration with OpenTelemetry-compatible agent harnesses (Think, Flue, AI SDK), and a dedicated Agents view in the dashboard for visualizing, debugging, and replaying agent sessions. Agent tracing complements existing Workers tracing by adding agent-specific spans and metadata.

6/12/2026

This post details the engineering efforts to scale Cloudflare's Security Insights scanning capacity by over 10x. Key technical contributions include: optimizing Kafka consumer processing by introducing batching and parallel goroutines, implementing a 'slow lane'/'fast lane' approach to mitigate head-of-line blocking, optimizing database inserts using a hybrid UNNEST/COPY strategy for Postgres, resolving API latency issues by switching to an active-passive database configuration, and re-architecting the scheduler to ensure uniform scan distribution through independent zone scheduling, randomized timing, and adaptive rate limiting.

3/10/2026

This post introduces a revamped Security Overview dashboard that shifts from providing raw data to actionable insights. It introduces 'Security Action Items' ranked by criticality, a 'Detection Tools' module to address the configuration gap, and deep-linking from Suspicious Activity cards to Security Analytics. The post details the engineering behind this, including a system of specialized 'checkers' for scheduled and real-time analysis, and the introduction of 'Contextual Insights' to provide business impact and technical root cause for security events, particularly highlighting enhanced DNS insights for dangling records and subdomain takeover risks.

3/10/2026

This post introduces the expansion of Cloudflare Log Explorer to integrate 14 new datasets, covering the full surface of Cloudflare's Application Services and Cloudflare One product portfolios. It details how correlating telemetry from application-layer HTTP requests, network-layer DDoS and Firewall logs, and Zero Trust Access events enables security analysts to reduce Mean Time to Detect (MTTD) and unmask sophisticated, multi-layered attacks. It also provides specific examples of how to use Log Explorer for security forensics, including identifying reconnaissance, diversions, approach, identity audits, and data exfiltration, and highlights the new Tabs feature for correlating data across multiple datasets.

3/5/2026

Introduced Dynamic Path MTU Discovery (PMTUD) to the Cloudflare One Client, leveraging RFC 8899 and the MASQUE protocol over Cloudflare's QUIC library. This allows the client to actively probe network paths for optimal packet sizes, overcoming PMTUD black holes caused by silent packet drops and network middleboxes. The implementation dynamically resizes the virtual interface MTU on the fly, ensuring stable and uninterrupted connections for users across diverse network conditions.

12/5/2025

This post details a critical incident where a change to the internal WAF tool's buffer size, intended to mitigate a React Server Components vulnerability, inadvertently caused widespread HTTP 500 errors due to a bug in the FL1 proxy's rules module. The bug, a nil value dereference in Lua when a rule with an 'execute' action was skipped, highlights a weakness in the language's type system and the need for more robust error handling and gradual rollout mechanisms. The incident underscores the importance of comprehensive testing for all system components, including internal tools, and the need for 'fail-open' error handling strategies. The post also references ongoing work from a previous outage on November 18, 2025, focusing on enhanced rollouts, streamlined break-glass capabilities, and 'fail-open' error handling.

11/18/2025

This post details the root cause and impact of the November 18, 2025 Cloudflare outage, which was triggered by a change in database permissions leading to an oversized feature file for the Bot Management system. It describes the failure of the core proxy (FL and FL2) due to this oversized file, the impact on various services like Turnstile, Workers KV, and Access, and the mitigation steps taken. It also highlights the importance of robust change management, feature file validation, and the interplay between different system components in maintaining network reliability.

10/28/2025

Introduces automatic tracing for Cloudflare Workers, enabling developers to gain deep visibility into application performance and errors without manual instrumentation. This feature captures OpenTelemetry-compliant spans for I/O operations (bindings, fetch, handlers), provides detailed attributes, and allows traces to be viewed in the dashboard or exported to third-party providers. It also introduces correlated log and trace exports and outlines future enhancements like trace context propagation and custom spans.

10/27/2025

This post details the technical implementation and methodology behind Cloudflare's Speed Test. It explains how the test uses Cloudflare Workers and anycast architecture to route users to the nearest data center. It highlights the test's focus on measuring network quality through predefined data blocks rather than saturating the connection, and its assessment of loaded latency, idle latency, jitter, and packet loss. The post also introduces the Aggregated Internet Measurement (AIM) score as a user-friendly metric for overall connection performance.

10/27/2025

This post details the evolution of Cloudflare Radar, launched in 2020, as a key component of Cloudflare's observability platform. It highlights how Radar has expanded to provide comprehensive insights into Internet security, routing, traffic disruptions, protocol adoption, DNS, and AI. Specific contributions include the launch of Certificate Transparency insights, visibility into TCP resets and timeouts for detecting connection tampering, tracking post-quantum encryption adoption, and measuring AI bot/crawler activity. The post emphasizes Radar's role in making the internet more observable, transparent, and resilient.

10/21/2025

This post details performance bottlenecks in the BPF LPM trie data structure, specifically focusing on lookup, update, and delete operations. It analyzes the limitations of the current two-child node implementation, the lack of level compression, and the impact of dense data distributions on lookup throughput. The post benchmarks performance metrics and identifies the root causes of slow operations, including L1 dcache miss rates, which are critical for network packet routing and security rule evaluation.

9/26/2025

Introduced Observatory, a new observability product that integrates real-user data, backend telemetry, error rates, cache hit ratios, and synthetic testing into a single platform for performance monitoring and analysis. Also launched Smart Shield, which provides one-click fixes for common performance issues identified by Observatory. Detailed the data sources used in Observatory, including RUM, backend telemetry, error rates, cache hit ratios, and synthetic testing (browser and network tests).

9/17/2025

Introduces the initial phase of a major upgrade to Cloudflare's performance analytics suite by combining Web Analytics with network-level insights. This will enable developers to pinpoint performance issues from the visitor's browser through Cloudflare's network to the origin, while preserving user privacy. Web Analytics will be enabled by default for all free domains on October 15, 2025, collecting client-side metrics like Core Web Vitals and resource load times. Data is aggregated at the edge to reduce storage and minimize latency, removing personal information. The platform aims to provide a privacy-first approach to performance monitoring and debugging.

9/13/2025

This post details an incident where a bug in the dashboard's React useEffect hook caused repeated, unnecessary calls to the Tenant Service API due to a problematic object in its dependency array. This, combined with a service update to the Tenant Service API, overwhelmed the service and led to a broad outage of APIs and the Cloudflare Dashboard. The response involved increasing resources for the Tenant Service and implementing a temporary rate limit. Going forward, Cloudflare will migrate the Tenant Service to Argo Rollouts for automatic rollback, introduce random delays to dashboard retries to mitigate thundering herd issues, allocate more capacity to the Tenant Service, and enhance API calls with retry/new request indicators for improved observability.

8/29/2025

Introduces two new AI-powered tools for Cloudflare One: a WARP diagnostic analyzer in the Zero Trust dashboard that uses AI to interpret WARP diagnostic logs, and an MCP (Model Context Protocol) server for Digital Experience Monitoring (DEX). The WARP diagnostic analyzer provides a summary of events, detailed explanations, and recommended troubleshooting steps. The DEX MCP server allows users to ask natural language questions about device performance and connectivity, generating customized graphs and insights without manual data pipeline integration.

8/5/2025

This post details the investigation and resolution of high latency (~40 ms) in double-spend checks for Cloudflare's privacy proxy product. The root cause was identified as the interaction between Nagle's algorithm and delayed ACKs in TCP, leading to inefficient packet transmission for small `memcached` commands. The fix involved using `BufWriter<TcpStream>` to buffer writes and send the complete `memcached` command in a single message, reducing latency to <1 ms.

6/18/2025

This post announces the General Availability of Cloudflare Log Explorer, a new product providing native observability and forensics within the Cloudflare dashboard. It highlights benefits such as reduced cost and complexity, faster detection and triage, accelerated investigations with full context, and minimal recovery time. Key features include custom dashboards with natural language chart creation and AI assistance, expert-curated dashboard templates, a streamlined Log Search with SQL interface, saved queries for collaboration, and upcoming custom alerting and flexible log retention. The post also details the underlying architecture of Log Explorer, consisting of ingestors, compactors, and queriers, built on Cloudflare R2 storage.

5/5/2025

This post details the implementation and optimization of Health Mediated Deployments (HMD), Cloudflare's data-driven system for automating software updates across its global network. It explains how HMD uses Prometheus metrics queried via Thanos to monitor service health (e.g., HTTP 500 error rates) and automatically revert problematic releases. The post also covers optimizations made to the backtesting process, including the use of recording rules for pre-computation and the adoption of Thanos' distributed query engine to improve query performance and reduce latency by processing data center-scoped aggregates remotely. The use of R2 for storing historical metric data and Cloudprober for measuring performance improvements are also highlighted.

4/9/2025

Introduces Workers Observability, a new dashboard section for querying detailed log events across all Workers. This includes a Workers Metrics Dashboard (Beta) for viewing metrics and logs from all Workers, a Query Builder (Beta) for structured log exploration, metric extraction, and visualization, and makes Workers Logs Generally Available with a public API and improved invocation-based grouping. It also adds CPU time and wall time monitoring for Workers invocations and revamps the Workers metrics overview page. A new Invocations View groups logs by execution event, and a Workers Observability API is introduced for programmatic access to telemetry data.

3/18/2025

This post introduces the Cloudforce One threat events platform, a new capability that leverages Cloudflare's network traffic insights to provide contextualized threat intelligence. It allows security practitioners to analyze detection hits from indicator feeds with rich context, moving beyond simple IOCs to include information on threat actor activity, tools, techniques, and procedures, mapped to frameworks like MITRE ATT&CK. The platform is built using Cloudflare Workers and SQLite-backed Durable Objects, offering a scalable and resilient solution for real-time threat analysis.

3/18/2025

This post announces the expansion of Log Explorer to include logs from Cloudflare's Zero Trust product suite (Access, Gateway DNS, Gateway HTTP, Gateway Network, CASB, Zero Trust Network Session, Device Posture Results). It also introduces custom dashboards that can be created using natural language prompts and pre-built templates for monitoring specific security and performance metrics. The integration allows for seamless transition from dashboard observation to investigation and mitigation within the Cloudflare platform.

2/13/2025

This post introduces Automatic Audit Logs, a significant enhancement to Cloudflare's observability capabilities. It standardizes and automates the generation of audit logs across 111 products (95% coverage), capturing a wider range of actions including GET requests and failed attempts. The system leverages the OpenAPI Schema for sensitive data redaction and streams requests via RPC to a dedicated Worker for minimal latency. This provides granular filtering, enhanced context, and improved transparency and accountability for user actions within the Cloudflare environment.

11/26/2024

This post details a specific incident involving the Cloudflare Logs system, highlighting failures in Logfwdr configuration and Buftee's overload protection. It provides a deep dive into the architecture of Logfwdr, Logreceiver, Buftee, and Logpush, explaining their roles in collecting and delivering customer event logs. The root causes of the incident are identified as a bug in the Logfwdr configuration system and a failure to configure Buftee's overload protection mechanisms. The post outlines future steps to prevent recurrence, including improved alerting, testing, and overload simulations.

10/31/2024

Baselime migrated its observability platform from AWS to Cloudflare, rebuilding its data pipeline on Cloudflare Workers, Durable Objects, and Workers Analytics Engine. This resulted in a simplified architecture, improved performance (lower latency), enhanced scalability and reliability, and over 80% cost reduction. Key technical changes include using JavaScript-native RPC for inter-component communication, leveraging Durable Objects for stateful processing (error tracking, pattern detection, alerting), and utilizing Workers Analytics Engine with ABR analytics for high-cardinality data analysis. The migration also involved transitioning from AWS EventBridge and SQS to Cloudflare Queues using the strangler fig pattern.

10/23/2024

This post introduces an anomaly detection pipeline for DDoS protection that builds statistical models to flag unusual traffic. It details the evolution from naive volumetric models to time series forecasting and finally to a multi-dimensional approach using Principal Component Analysis (PCA) and Mahalanobis distance to identify novel DDoS attacks by analyzing deviations in various traffic features beyond just volume.

10/10/2024

Introduces DEX notifications for proactive monitoring within Cloudflare One. These notifications allow administrators to create configurable alert rules based on z-score and SLOs for device connectivity anomalies, test latency (HTTP and Traceroute tests), and test low availability. The post details the problem solved, the statistical methods used (z-score), and provides examples of notification configurations for different scenarios.

10/9/2024

This post introduces Cloudflare's development of an automatic remediation system built on Temporal workflows to enhance platform resilience. It details the creation of a coordinator for authorization and scheduling, task routing for efficient execution across the global network, and various trigger mechanisms (alerting systems, log tailers, health checks, chatbots) to initiate self-healing workflows. This significantly reduces manual intervention ('toil') and improves Mean-Time-To-Resolve (MTTR) for system failures, even before they impact users.

9/5/2024

This post introduces a new capability to the Observability Platform by launching a dashboard and API endpoint on Cloudflare Radar that provides near real-time insights into anomalous TCP connections (resets and timeouts) to Cloudflare's network. It details the classification of these anomalous connections, their potential sources (scanners, application shutdowns, network errors, attacks, tampering), and provides guidance on interpreting the data for self-evaluation, exploring new targets, and longitudinal studies. This significantly enhances the platform's ability to diagnose and understand network-level issues.

7/22/2024

This post analyzes pre-event internet trends and security insights related to the Paris 2024 Olympics. It leverages Cloudflare's 1.1.1.1 resolver to track DNS traffic to official Olympic websites, identifying France as the leading country in interest. It also analyzes spikes in DNS traffic related to ticket sales and major announcements, and examines the rise of Olympics-related spam and malicious emails processed by Cloudflare's Email Security service. This contributes to the observability platform by providing real-world data on user interest and security threats surrounding a major global event.

6/26/2024

This post details a specific incident on June 20, 2024, caused by two independent events: suboptimal network traffic re-routing due to automated monitoring and a latent bug in the rate limiting system triggered by a new DDoS mitigation mechanism. The latter event, where a specific HTTP request caused a process to enter an infinite loop, is described in technical detail, including the root cause involving a recursive function call due to an anticipated internal API usage and a broken cookie validation function. The post also outlines the incident timeline, impact metrics (error rates, TTFB latency), and the mitigation steps taken, such as disabling the DDoS rule and restarting services.

6/21/2024

This post demonstrates how Cloudflare's global network visibility, a core component of its observability platform, can be leveraged to analyze real-world internet traffic trends during major events like the UEFA Euro 2024. It provides data on traffic drops during games, identifies specific countries impacted, and details observed application-layer DDoS attacks, showcasing the platform's ability to provide insights into network behavior and security incidents during high-profile events.

6/10/2024

This post analyzes internet traffic trends and cybersecurity insights during the 2024 European Parliament election. It details traffic drops during voting hours, increases as results were announced, and specific impacts of national announcements in Belgium and France. It also highlights increased DNS traffic to election-related sites and news outlets, and reports on cyberattack activity targeting government-related websites.

6/7/2024

This post introduces the Cloudflare Radar platform as a tool for tracking and analyzing internet traffic trends during major global events, specifically focusing on national elections in the Netherlands, South Africa, Iceland, India, and Mexico. It details observed patterns of internet traffic decreases during polling hours and increases as results are announced, and highlights instances of cyberattacks, particularly DDoS attacks targeting political websites during the European Parliament election. The post also provides country-specific insights into traffic shifts and attack trends during the elections in Mexico, India, and the Netherlands.

6/3/2024

This post details the migration of Cloudflare's large-scale logging pipeline from syslog-ng to the OpenTelemetry Collector. It covers the motivations for the shift, including the use of Go for easier contributions, improved build chains with Post-Quantum cryptography, enhanced telemetry via Prometheus metrics, and unification with existing OpenTelemetry Collector usage for tracing. The post describes the development of four internal OpenTelemetry Collector components: `cfjs1exporter` for custom log formatting, `fileexporter` with custom permissions and formats, `externaljsonprocessor` for enriching logs with external data, and `ratelimit processor` for service-level rate limiting. It also outlines the migration process for edge and core data centers, including the use of Salt states and manual onboarding for custom configurations. Lessons learned include the importance of robust failover mechanisms for central logging servers and the need for more granular instrumentation around niche services and internet 'badness'.

5/14/2024

This post details the Observability Team's successful implementation of Profile Guided Optimization (PGO) for their Go-based service 'wshim'. It outlines the process of collecting production CPU profiles, merging them, and recompiling the service with PGO enabled. The results show a ~3.5% reduction in CPU usage, saving approximately 97 cores globally, demonstrating significant resource reclamation without code changes.

4/5/2024

Cloudflare has acquired Baselime, a company specializing in serverless application observability. This acquisition aims to integrate Baselime's capabilities, including high cardinality and dimensionality data handling, OpenTelemetry native support, and a focus on developer experience, into Cloudflare's existing observability platform. The integration will enable deeper insights into serverless applications built on Cloudflare's platform, such as Workers, D1, R2, and Durable Objects, with potential for embedding OpenTelemetry directly into the Workers runtime.

3/29/2024

This post details the evolution of Cloudflare's alert observability by implementing a new data pipeline using Vector.dev to aggregate all alert states (firing, silenced, inhibited, resolved) from Prometheus Alertmanager into ClickHouse. This addresses limitations of previous tools by providing complete state information, enabling better troubleshooting, reporting, and analysis of alert noise to mitigate on-call burnout. New dashboards for alert overview, alertname breakdown, receiver-specific insights, state timelines, and silence analysis have been developed.

3/8/2024

Cloudforce One has built a threat investigation platform using Cloudflare Workers, Pages, R2, KV, D1, and Vectorize. This platform manages Requests for Information (RFIs) and Priority Intelligence Requirements (PIRs) for threat intelligence customers. The architecture includes services for RFIs/PIRs, a UI via Pages, a case management system using Durable Objects and WebSockets, a queue for informal requests (Leads) stored in KV and D1, and a binary database in R2. The system leverages Vectorize for searching indexed case files, enabling real-time collaboration and advanced threat analysis.

3/8/2024

This post introduces the URL Scanner, a new tool within Cloudflare Radar that detects and safeguards against malicious websites. It details the integration of the URL Scanner with the Security Center in the Cloudflare Dashboard, enhancing cybersecurity workflows. The post also highlights the new URL Scanner API, enabling custom scans with features like custom HTTP headers and multi-device screenshots, and explains the underlying technology stack, emphasizing the use of Cloudflare Workers, Durable Objects, and the Browser Rendering API.

3/8/2024

This post introduces Log Explorer, a new feature that extends the existing Security Analytics offering. Log Explorer allows users to directly investigate raw HTTP and Security Event logs within the Cloudflare Dashboard, eliminating the need for third-party storage. This provides granular visibility for SOC analysts and compliance officers, enabling them to troubleshoot issues, investigate security incidents, and tune performance and security by analyzing specific log events. It leverages Cloudflare R2 and Delta Lake for cost-effective, performant, and ACID-compliant log storage.

3/7/2024

Introduces a free version of Magic Network Monitoring for all Enterprise Customers, enabling them to send NetFlow or sFlow data to Cloudflare for network traffic analysis. Details how this product was used internally during the Thanksgiving 2023 security incident and outlines use cases for monitoring local IoT and cloud network traffic. Provides instructions for Enterprise Customers to gain access and mentions future support for native cloud VPC flow logs.

2/7/2024

This post announces updates to Cloudflare One, the SASE platform, focusing on network modernization to fulfill the promise of single-vendor SASE. It introduces more flexible on-ramps for site-to-site connectivity (agent/proxy-based and appliance/routing-based), WAN-as-a-service (WANaaS) capabilities (high availability, application awareness, VM deployment, enhanced visibility/analytics), and Zero Trust connectivity for DevOps (mesh and peer-to-peer secure networking). This enhances the platform's ability to provide unified network and security visibility and control for various teams.

1/24/2024

This post introduces 'Foundations,' an open-source Rust service foundation library designed to simplify the development of distributed, production-grade systems. It addresses key challenges in production environments: observability (logging, tracing, metrics), configuration, and security. The library provides a unified API for telemetry, including enhanced trace sampling flexibility, distributed trace stitching, and trace forking. It also offers implicit context propagation for logging and integrates the Prometheus Rust client library for metrics with a procedural macro for simplified metric definition.

1/8/2024

This post details the architecture and operation of Cloudflare's internal logging pipeline, which ships close to a million log lines per second. It covers the journey of logs from service stdout/stderr to systemd-journald, then to syslog-ng for rate limiting and enrichment, and finally to two core data centers (log-a and log-b) for redundancy via Kafka. The post also discusses challenges in log ordering due to syslog limitations, partitioning strategies for Kafka, and long-term storage solutions using Elasticsearch and ClickHouse. Future work includes migrating to Open Telemetry Logs, tail sampling, and improving Kafka cluster balancing.

11/4/2023

This post details a major outage of Cloudflare's control plane and analytics services caused by a power failure at a third-party data center. It highlights the failure of a critical data center provider, the unexpected behavior of backup power systems, and the lack of communication from the provider. Crucially, it reveals that while the core network remained operational, critical control plane and analytics systems (Kafka, ClickHouse) were not fully integrated into the high availability cluster and had non-obvious dependencies on the failed data center, leading to extended downtime. The post outlines intended design for data center failure, what failed (dependencies on PDX-04, incomplete HA cluster migration, lax GA requirements for new products), what worked (core network, disaster recovery facility for some services), and proposed changes to ensure all critical systems are in the HA cluster and dependencies are better managed.

10/31/2023

Introduced Traffic Anomalies notifications for enterprise customers, which trigger based on unexpected changes in HTTP traffic. This feature utilizes z-scores to compare traffic over the past 5 minutes against the average over the past 4 hours, alerting when the deviation exceeds 3.5 standard deviations. Minimum thresholds of 200 requests for spikes and drops are applied to reduce noise. The system stores sampled HTTP traffic statistics in Clickhouse, then moves them to VictoriaMetrics for time-series analysis, leveraging its algorithmic functions and scalability. The existing Alert Notification System is used for delivery.

10/18/2023

This post announces the general availability of Magic Network Monitoring (MNM), a cloud network flow monitor. It details the technical challenges and solutions involved in improving traffic volume estimations in MNM, specifically the transition from simple random sampling to VarOpt reservoir sampling to accurately handle the long-tail distribution of network flow data weights. It also outlines how MNM integrates with Magic Transit and Magic WAN for enhanced DDoS detection and network visibility.

9/26/2023

This post introduces the integration of real-time traffic anomaly detection, route leak, and route hijack insights into Cloudflare Radar. It also launches a new notification functionality allowing users to subscribe to these events via email or webhooks, enhancing proactive monitoring and awareness of internet disruptions.

9/26/2023

This post details how Cloudflare Radar detects Internet outages by analyzing traffic anomalies across various data sources (DNS, HTTP, NetFlows, NEL). It introduces the 'Traffic anomalies' table in the Outage Center, with statuses like 'Unverified' and 'False Positive', and the ability for users to subscribe to notifications for country and AS-level events. The post also illustrates the complexity of modeling internet traffic patterns and the challenges in detecting anomalies across diverse scenarios, highlighting the need for robust preprocessing and statistical approaches.

9/25/2023

Introduced Incident Alerts, a new proactive notification system accessible via email, webhook, or PagerDuty. These alerts are customizable by product and impact level, offering a push notification model to complement existing pull-based methods like the Status API and RSS feeds. This enhances the observability platform by providing direct, configurable communication channels for service disruptions.

9/25/2023

Introduced Cloudflare Trace, a new product that provides detailed visibility into how HTTP requests traverse Cloudflare's configuration and which rules are applied. This feature allows users to understand the journey of their traffic through Cloudflare's complex ecosystem, debug issues related to rule application, and export trace data for support tickets. The development of Trace was informed by internal tools like Flute and customer feedback, aiming to strike a balance between detail and usability.

9/21/2023

This post announces an integration between Cloudflare Email Security and CrowdStrike Falcon LogScale, allowing joint customers to send and visualize email security detection data within Falcon LogScale. It details the setup process and highlights the benefits of real-time search, dashboards, and visualizations for threat hunting. The post also outlines future plans to extend this integration to other Cloudflare Zero Trust products.

9/20/2023

This post details the evolution of Cloudflare's Waiting Room product, focusing on its core mechanism for queueing traffic during spikes. It explains how Waiting Room leverages Cloudflare Workers and Durable Objects for distributed decision-making to minimize latency. The post elaborates on the configuration options (total active users, new users per minute, session duration) and the algorithm used at the worker level to decide when to queue users, considering traffic history and distribution across data centers and workers to optimize user experience and prevent origin overload.

9/1/2023

The Workers Tail feature has been significantly scaled to support Workers of any size. This was achieved by moving log filtering logic from the Durable Object to the message producer, preventing unnecessary load. A sampling mechanism was introduced to keep Durable Object RPS below a target of 100, with a visual indicator for users. A failsafe mechanism was also added to periodically stop forwarding messages if the Durable Object becomes overloaded.

8/3/2023

This post announces the general availability of the Cloudflare Zero Trust Integration with Datadog, which includes a Cloudflare Content Pack within Datadog Cloud SIEM. This integration provides out-of-the-box dashboards and detection rules for customers ingesting Zero Trust logs into Datadog, offering improved security insights. Specific features include enhanced security insights over the Zero Trust landscape, security detections built for CASB findings, and security detections for impossible travel scenarios. The integration facilitates log ingestion via Logpush jobs and installation of the Cloudflare Tile in Datadog.

7/11/2023

This post details the evolution of the `ping` utility, a fundamental network diagnostic tool, highlighting its 'countermeasures' against clock manipulation. It explores how `ping` handles time synchronization issues, the underlying mechanisms like `vdso` and `CAP_NET_RAW` capabilities, and the use of `strace` for debugging. This contributes to the broader observability platform by demonstrating how even basic network tools are being refined for greater accuracy and reliability in performance measurement.

6/26/2023

This post announces the launch of Observatory, Digital Experience Monitoring (DEM) beta, and Timing Insights during Speed Week 2023. Observatory provides performance metrics and recommendations. DEM offers insights into application performance and user experience through Synthetic Application Monitoring and Fleet Status. Timing Insights provides new performance metrics via the GraphQL API, focusing on understanding Time To First Byte (TTFB) and its components.

6/22/2023

This post introduces the open beta of Cloudflare's Digital Experience Monitoring (DEX) product, which is part of the Cloudflare One SASE offering. DEX provides tools for measuring end-user connectivity and performance, including Synthetic Application Monitoring (SAM) for testing public or private applications and Fleet Status for real-time insights into WARP-enrolled devices. SAM allows administrators to schedule HTTP GET tests to monitor application response times from various global locations, visualize resource fetch times, and analyze server/DNS response times and HTTP status codes over time. Fleet Status provides global and per-device views of device connection status, mode, and location, aiding in troubleshooting connectivity issues.

6/20/2023

Introduces 'Timing Insights' via the GraphQL Analytics API, providing new metrics to break down Time to First Byte (TTFB). New fields include `edgeTimeToFirstByteMs`, `edgeDnsResponseTimeMs`, and `originResponseDurationMs`, available at various quantiles (P50, P95, P99). Demonstrates how to use these metrics to diagnose performance bottlenecks, specifically highlighting DNS resolution as a potential cause for slow TTFB.

5/2/2023

This post details the redesign of Cloudflare's Network Analytics system to version 2, addressing limitations of the v1 system by incorporating stateful mitigation data from systems like flowtrackd. The new design adopts a Netflow/sFlow-inspired approach, where individual software components (network functions) emit metadata-rich packet samples. This enables a more comprehensive and future-proof observability system. The post also discusses the implementation challenges of the new data pipeline, 'samplerd', and the modifications made to existing systems like xdpd (specifically l4drop) to emit these richer samples.

3/14/2023

This post details the integration of Cloudflare's Zero Trust logs (from Gateway, Remote Browser Isolation, Data Loss Prevention, Access, and Cloud Access Security Broker) with Sumo Logic's Cloud SIEM. It describes how this integration enables automated normalization and correlation of security signals, reducing alert fatigue and accelerating security analyst triage by converging security and network data into high-fidelity insights. It also mentions the existing Cloudflare App for Sumo Logic and outlines the steps for customers to enable Logpush to Sumo Logic and install the Cloudflare App for enhanced Zero Trust visibility.

3/3/2023

This post details the challenges and strategies for operating Prometheus at scale, specifically addressing metrics cardinality and memory management for a deployment of 916 instances handling billions of time series. It explains the concepts of metrics, samples, and time series, the impact of labels on cardinality, and how Prometheus uses memory during the scrape and TSDB ingestion process. It also touches upon strategies for dealing with cardinality-related problems.

2/3/2023

Introduced advanced HTTP alerting capabilities for Enterprise customers, allowing filtering and organization of alerts based on origin response status codes, edge response status codes, alert sensitivity/SLO, client IPv4/IPv6 addresses, and specific zones. This enhances the observability platform by providing more granular control over monitoring schemes.

1/9/2023

Introduces Digital Experience Monitoring (DEM) as a new product within the observability platform. Key components include Zero Trust Fleet Status for device agent monitoring, Synthetic Application Monitoring for proactive testing of public-facing endpoints via traceroute and HTTP GET requests, and Network Path Visualization for hop-by-hop telemetry data. These features aim to provide administrators with granular visibility into end-user connectivity and performance issues.

11/18/2022

Introduced Workers Trace Events Logpush, a feature that allows developers to send Cloudflare Workers logs to external destinations like object storage or analytics platforms. This extends the existing `wrangler tail` functionality by providing historical logs and enabling monitoring. The post details the log format, pricing, and configuration steps, including an example using curl to create a Logpush job and enabling logging in `wrangler.toml`.

9/28/2022

This post announces the early access release of a free version of Magic Network Monitoring (MNM), a network flow analytics tool. It details the features of MNM, including an analytics dashboard with filtering capabilities (protocol, source/destination IP, TCP flags, router IP), network traffic volume alerts per IP address or prefix, and its integration with Magic Transit for DDoS mitigation. The post provides use cases for monitoring home labs and identifying network layer DDoS attacks, and outlines future planned features such as advanced DDoS identification and alert recommendations.

6/21/2022

This post announces and describes Cloudflare One Observability, a new capability designed to unify data from various Cloudflare One functions into a single experience for troubleshooting network connectivity, security policies, and performance issues. It highlights how this will simplify troubleshooting for bandwidth issues, network vulnerabilities, and slow network performance by automating data collection and aggregation, leveraging Cloudflare's integrated platform and single-pass inspection architecture. It also mentions the use of Instant Logs and ABR for data analysis.

5/19/2022

Introduced a new open-source tool, 'prom-rule-validator', to validate Prometheus alert rules. This tool helps prevent silent failures in alerting by checking for common issues like typos, stale metrics, and incorrect range query durations that can lead to alerts not firing when expected. The tool analyzes alert rule definitions and simulates Prometheus query behavior to identify potential problems before deployment.

5/10/2022

This post announces the addition of Workers execution logs to Logpush for Enterprise customers. It details the inclusion of unstructured console.log messages, exceptions, and metadata about requests/responses in the 'Trace Events' dataset. A sample Trace Event JSON is provided, showcasing the structure of the data. The post highlights how this enables customers to send these logs to external storage (GCS, R2) or analysis platforms (Splunk, New Relic) for debugging, performance analysis, and SaaS customer visibility.

3/18/2022

This post outlines the future vision for Cloudflare observability, defining it as having three core components: monitoring, analytics, and forensics. It details plans to enhance notification algorithms for accuracy and reduce noise, improve dashboard customization for analytics, and expand log access through Logpull, Logpush, and Instant Logs, with a focus on building log storage on R2. The post also introduces tracing as a method to understand the end-to-end life cycle of a request, differentiating it from logs by capturing context and enabling debugging of individual operations within complex applications like Workers.

3/17/2022

Introduced on-demand packet captures from Cloudflare's global network for customers using Magic Transit and Magic WAN. This feature allows users to capture packet data via an API, filtering by IP address, ports, and protocol. The implementation uses nftables for filtering and nflog for logging packets, with tcpdump used in user space to generate pcap files. Packet capture files are delivered directly to customer-chosen cloud storage, ensuring data privacy.

3/14/2022

This post introduces a direct integration between Cloudflare's Logpush product and New Relic, a third-party observability platform. This integration aims to simplify the process for customers to ingest and correlate Cloudflare logs with their existing telemetry data in New Relic One, eliminating the need for middleware and reducing costs. It highlights a new quickstart for New Relic that provides a pre-built dashboard for monitoring and analyzing web traffic metrics from Cloudflare logs.

12/11/2021

This post details the evolution of Cloudflare's Notifications platform, introducing webhook support for direct integration with various messaging and monitoring systems (Slack, Teams, Discord, DataDog, Splunk, OpsGenie) and the addition of notification history for the past 30-90 days, accessible via API. This enhances the observability of system events by providing more direct and persistent channels for alerts.

9/18/2021

Introduces beta support for local CPU profiling of Cloudflare Workers using Wrangler and integration with Chrome DevTools protocol. This allows developers to inspect and analyze the performance of their Workers by gathering stack traces at a sampled rate, providing insights into CPU-intensive functions and potential bottlenecks. The feature leverages `wrangler dev --inspect` and the `chrome://inspect` interface to visualize flame graphs and annotated source code.

9/16/2021

Introduced Last Mile Insights, a new feature that provides customers with visibility into end-user connection issues on the last mile. This feature leverages Network Error Logging (NEL) and machine learning to detect and diagnose problems, allowing customers to see where their end-users are having trouble connecting to Cloudflare properties, where it failed, and why. This is a significant addition to Cloudflare's observability capabilities, enabling proactive issue detection and faster resolution.

9/14/2021

This post details the architecture and implementation of 'Instant Logs', a new system for real-time access to Cloudflare HTTP logs. It leverages Cloudflare Workers for edge computation and Durable Objects for coordination and buffering, enabling sub-three-second latency. The system employs a MapReduce-like pattern with filtering Workers and sharded Durable Objects to handle varying traffic loads. Reservoir sampling is used for controlled data shedding at high volumes, preserving statistical properties of the original traffic.

4/13/2021

This post announces new partnerships with six observability-focused companies (Sentry, New Relic, Datadog, Splunk, Sumo Logic, and Honeycomb.io) that integrate with Cloudflare Workers. It details how to use Sentry with the `toucan-js` npm module for error tracking and performance monitoring, and how to send logs to New Relic using `event.waitUntil()` and their HTTP log endpoint. It also explains how to ingest Cloudflare Workers logs into Datadog, Splunk, and Sumo Logic via their respective HTTP APIs, and highlights Honeycomb.io's event-based model for application telemetry. Specific Workers fields like WorkerCPUTime, WorkerStatus, WorkerSubrequest, and WorkerSubrequestCount have been added to HTTP logs for enhanced visibility.

3/12/2021

This post details the evolution of Cloudflare's Anomaly Detection platform for Bot Management. It describes the use of Histogram-Based Outlier Scoring (HBOS) with ClickHouse for baselines and Redis with HyperLogLog for visitor behavior. The platform architecture is a series of microservices on Kubernetes, processing 500K requests/sec. Early challenges with a monolithic service and Redis PFMERGE were addressed by optimizing Redis key encoding and introducing a recency register. The platform was then refactored into microservices, including a dedicated Baseline service to reduce ClickHouse load and leveraging existing inserter code for data persistence.

1/14/2021

This post introduces SOAR (Simulation for Observability, reliAbility, and secuRity), a dedicated data center environment for running simulations of Cloudflare's production software stack. SOAR aims to address the challenges of increasing software complexity and the need for rigorous testing before deploying changes to production. It allows for isolated, coordinated, and engineer-friendly simulations of various products, including Magic Transit, by simulating end-user, product, and origin servers within a controlled LAN environment. This enables testing of functional correctness, performance metrics, and resource usage, as well as complex scenarios like route failover that were not possible with previous static simulation systems. The architecture includes an internal coordinator service for interfacing with engineers and dispatching tasks, and simulation agents on each server to execute these tasks.