TechPrufer
Get started
Cloudflare
Cloudflare logoCloudflare · Feature trail

Observability Platform

87 posts · Apr 2013 – Oct 2026

What this trail covers

Cloudflare's observability platform has evolved to unify logs, traces, analytics, alerts, dashboards, and exporting into a single, integrated experience. This includes a unified SQL API for querying diverse data sources (HTTP events, firewall events, Workers, Containers, R2, AI Gateway, analytics), end-to-end tracing from edge to origin, custom alerts based on SQL queries or anomalies, 30 days of domain analytics retention, and Logpush availability on all self-serve plans. The platform aims for comprehensive debugging and performance analysis, now including on-demand CPU and memory profiling for Workers and Durable Objects via flamegraphs, enabling detailed inspection of function-level resource usage in production.

All Cloudflare trails
  1. Introducing on-demand CPU and memory profiling with flamegraphs for Workers and Durable ObjectsIntroduces on-demand CPU and memory profiling for Cloudflare Workers and Durable Objects, accessible via the CLI and dashboard. This feature allows engineers to capture and inspect interactive flamegraphs of production execution, providing detailed insights into function-level CPU time and memory allocations. The post details the internal implementation challenges, such as locating active isolates and Durable Object primaries, and provides examples of how this profiling capability was used internally to identify and fix performance bottlenecks (e.g., recursive JSON replacer, duplicate metric calls) and memory issues (e.g., partially disabled Prometheus instrumentation). It also explains the process of profiling an isolate without stopping the Worker, holding the isolate lock only for profiler lifecycle operations.
  2. 8 major updates to Cloudflare ObservabilityThis post introduces eight major updates to Cloudflare's observability platform: unified log investigation across multiple datasets (HTTP, firewall, Workers, Containers, R2, AI Gateway) with SQL and natural language querying; open beta for end-to-end tracing from edge to origin with sampling rates, trace rules, and OpenTelemetry export; a unified SQL API for agents and Workers to query all observability data; new unified pricing based on ingestion and storage volume; custom alerts on any observability data using SQL or anomaly detection; 30 days of retention for domain analytics; custom dashboards built from observability data; and Logpush availability on all self-serve plans.
  3. Introducing Cloudflare Traces: follow requests through our entire platformThis post introduces Cloudflare Traces, a new open beta feature that extends automatic tracing beyond Workers to the entire request path across Cloudflare's platform. It enables users to see supported security rules, transformations, cache decisions, routing, Worker execution, and origin handling within a single request-level timeline. Key capabilities include automatic span generation for supported platform operations, control over tracing via sampling rates and Trace Rules, end-to-end context propagation using W3C traceparent headers, in-dashboard trace investigation, and export of traces via OpenTelemetry Protocol (OTLP) to compatible observability platforms. This significantly enhances end-to-end visibility for debugging and understanding request behavior across Cloudflare's infrastructure and customer stacks.
  4. How Cloudflare enforces engineering standards using AIThis post introduces the Cloudflare Codex, a governed set of engineering standards designed to be consumed by both humans and AI agents. It details the Codex's organization into domains, its RFC-based format with SHOULD/MUST keywords, and its lifecycle states (approved/enforced). The post explains how the Codex supports engineering workflows through consumers like the AI code reviewer (which has flagged 230,000+ violations), the spec reviewer (which has evaluated ~600 technical designs), and the incident report reviewer (which assesses postmortems). It also highlights the development of language-specific linters and a local CLI for the AI code reviewer to improve developer experience. The Codex is built on the Developer Platform, utilizing Workers, D1, and AI Gateway.
  5. Introducing: Cloudflare AgentsThis post introduces agent tracing as a new capability within Cloudflare's observability platform. It details how agent-specific telemetry, including model calls, tool executions, and token usage, is captured and visualized alongside existing Workers infrastructure traces. The new Agents view in the dashboard provides session replay and trace waterfall visualizations. Support is added for Think, Flue, and AI SDK agent harnesses, with future plans for direct OpenTelemetry ingestion.
  7. Your agent can now debug Workers with local tracingIntroduces automatic OpenTelemetry trace capture for local Worker invocations when an AI coding agent is detected during `wrangler dev` or `vite dev` sessions. Agents can now query these traces and correlated logs via the Local Explorer API to debug issues, identify failures (e.g., D1 schema mismatch), and verify fixes locally without deploying code. This leverages existing Workers Tracing instrumentation and the Miniflare runtime's ability to collect and store telemetry locally.
  8. Scaling Security Insights: how we achieved a 10x increase in global scanning capacityThis post details the technical challenges and solutions implemented to scale the Security Insights scanning capacity by over 10x. Key contributions include: optimizing Kafka consumer processing by introducing batching and parallel goroutines, implementing a 'slow lane'/'fast lane' system to avoid head-of-line blocking, optimizing Postgres inserts using a hybrid UNNEST/COPY approach, resolving API latency issues by switching to an active-passive configuration, and redesigning the scheduler with independent zone scheduling, randomized timing, and adaptive rate limiting to ensure uniform scan distribution.
  9. Building a security overview dashboard for actionable insightsThis post introduces a revamped Security Overview dashboard that shifts from providing raw data to actionable insights. It introduces 'Security Action Items' ranked by criticality, a 'Detection Tools' module to address the configuration gap, and deep-linking from Suspicious Activity cards to Security Analytics. The post details the engineering behind this, including a system of specialized 'checkers' for scheduled and real-time analysis, and the introduction of 'Contextual Insights' to provide business impact and technical root cause for security events, particularly highlighting enhanced DNS insights for dangling records and subdomain takeover risks.
  10. Investigating multi-vector attacks in Log ExplorerThis post introduces the expansion of Cloudflare Log Explorer to include 14 new datasets, covering the full surface of Cloudflare's Application Services and Cloudflare One product portfolios. It details how correlating telemetry from application-layer HTTP requests, network-layer DDoS and Firewall logs, and Zero Trust Access events allows security analysts to reduce Mean Time to Detect (MTTD) and unmask sophisticated, multi-layered attacks. Specific examples are provided for identifying reconnaissance (e.g., querying http_requests for specific status codes and paths, or magic_ids_detections for scans), diversions (network_analytics_logs for volumetric attacks), approaches (http_requests and fw_events for malicious payloads), auditing identity (access_requests), and stopping data exfiltration (DNS tunneling via QueryName and QueryType in DNS logs, or protocol anomalies in magic_ids_detections). The post also highlights the new Tabs feature for working with multiple queries simultaneously and pivoting between them.
  11. Ending the "silent drop": how Dynamic Path MTU Discovery makes the Cloudflare One Client more resilientIntroduced Dynamic Path MTU Discovery (PMTUD) to the Cloudflare One Client, leveraging RFC 8899 and the MASQUE protocol over Cloudflare's QUIC library. This allows the client to actively probe network paths for optimal packet sizes, overcoming PMTUD black holes caused by silent packet drops and network middleboxes. The implementation dynamically resizes the virtual interface MTU on the fly, ensuring stable and uninterrupted connections for users across diverse network conditions.
  12. Code Orange: Fail Small — our resilience plan following recent incidentsThis post details Cloudflare's 'Code Orange: Fail Small' resilience plan following recent network incidents. The plan focuses on three key areas: 1) Implementing controlled rollouts for all configuration changes, mirroring existing software release processes, to prevent rapid propagation of errors. 2) Reviewing and improving failure modes between services to ensure graceful handling of errors and well-defined behavior. 3) Revising 'break glass' procedures and removing circular dependencies to enable faster incident resolution. The post highlights the need to treat configuration changes with the same caution as software releases and details the Health Mediated Deployment (HMD) system's application to configuration management.
  13. Code Orange: Fail Small — our resilience plan following recent incidentsThis post details Cloudflare's 'Code Orange: Fail Small' resilience plan following recent network incidents. The plan introduces controlled rollouts for configuration changes, similar to software releases, using the Health Mediated Deployment (HMD) system. It also focuses on reviewing and improving failure modes between services, assuming failure will occur at interface contracts, and enhancing 'break glass' procedures to improve emergency response times by removing circular dependencies and ensuring faster access to tools. The immediate goals include ensuring all production systems are covered by HMD for configuration management, updating systems to adhere to proper failure modes, and establishing processes for emergency remediation access.
  14. The 2025 Cloudflare Radar Year in Review- the rise of AI, post-quantum, and record-breaking DDoS attacksThis post details the 2025 Cloudflare Radar Year in Review, highlighting trends in AI, post-quantum encryption adoption, and record-breaking DDoS attacks. It leverages data from Cloudflare's global network to provide insights into traffic, bots, security, connectivity, and DNS patterns. Key findings include a 19% growth in global internet traffic, a 15x increase in AI 'user action' crawling, and significant advancements in routing security. The post emphasizes the use of Cloudflare Radar for observing and analyzing these internet trends.
  15. Cloudflare outage on December 5, 2025This post details a critical incident where a change to the internal WAF tool's buffer size, intended to mitigate a React Server Components vulnerability, inadvertently caused widespread HTTP 500 errors due to a bug in the FL1 proxy's rules module. The bug, a nil value dereference in Lua when a rule with an 'execute' action was skipped, highlights a weakness in the language's type system and the need for more robust error handling and gradual rollout mechanisms. The incident underscores the importance of comprehensive testing for all system components, including internal tools, and the need for 'fail-open' error handling strategies. The post also references ongoing work from a previous outage on November 18, 2025, focusing on enhanced rollouts, streamlined break-glass capabilities, and 'fail-open' error handling.
  16. Cloudflare outage on November 18, 2025This post details a major outage caused by a database permission change leading to a doubled feature file size for the Bot Management system. The oversized file caused failures in the core proxy and downstream services. The incident highlights a gap in observability regarding the propagation of configuration files and the impact of database changes on critical systems. The resolution involved stopping the bad file propagation, manually inserting a good file, and restarting core proxies. The post also touches on the different impacts on FL and FL2 proxy engines and the coincidence of the status page going down.
  17. Finding the grain of sand in a heap of SaltThis post details the development of an automated root cause analysis system for Salt configuration management failures. It describes the architectural problem of job results living on Salt Masters, the solution of caching job results and inputs on minions, and the correlation of failures with git commits, external service failures, and ad hoc releases. This system reduces the duration of software release delays and the toil of repetitive triage for SREs.
  18. Fresh insights from old data: corroborating reports of Turkmenistan IP unblocking and firewall testingThis post details the use of Cloudflare Radar's TCP reset and timeout data to corroborate reports of large-scale network blocking and firewall testing in Turkmenistan. It analyzes traffic patterns and connection anomalies over time, correlating them with news events and specific Autonomous Systems (AS) within the country. The post highlights the limitations of passive observation and the importance of combining different data sources for insightful analysis.
  19. Announcing Workers automatic tracing -- now in open betaThis post introduces automatic tracing for Cloudflare Workers, available in open beta. It enables automatic instrumentation of I/O operations (binding calls, fetch calls, handler calls) within Workers, capturing OpenTelemetry-compliant spans with automatic attributes. Traces can be viewed in the Cloudflare dashboard or exported to OTLP providers. This expands the existing observability platform by adding a crucial layer for debugging and performance analysis of serverless applications.
  20. How does Cloudflare’s Speed Test really work?This post details the technical implementation and methodology behind Cloudflare's Speed Test. It explains how the test uses Cloudflare Workers and anycast architecture to route users to the nearest data center. It highlights the test's focus on measuring network quality through predefined data blocks rather than saturating the connection, and its assessment of loaded latency, idle latency, jitter, and packet loss. The post also introduces the Aggregated Internet Measurement (AIM) score as a user-friendly metric for overall connection performance.
  21. Making the Internet observable- the evolution of Cloudflare RadarThis post details the evolution of Cloudflare Radar into a comprehensive internet observability platform. It highlights the integration of data sets from Cloudflare Research projects, including Certificate Transparency (CT) insights, detection of connection tampering (TCP resets and timeouts), and post-quantum encryption adoption tracking. It also introduces AI Insights, providing visibility into AI bot and crawler activity, including traffic trends by bot and crawl purpose, crawl-to-refer ratios, and analysis of robots.txt files. Furthermore, the post discusses improvements in routing visibility, with a focus on detecting route leaks and origin hijacks, and the expansion of Radar's capabilities to include insights into network layer security and adoption of new technologies.
  22. A deep dive into BPF LPM trie performance and optimizationThis post details performance bottlenecks in the BPF LPM trie data structure, specifically focusing on lookup, update, and delete operations. It analyzes the limitations of the current two-child node implementation, the lack of level compression, and the impact of dense data distributions on lookup throughput. The post benchmarks performance metrics and identifies the root causes of slow operations, including L1 dcache miss rates, which are critical for network packet routing and security rule evaluation.
  23. Introducing Observatory and Smart Shield — see how the world sees your website, and make it faster in one clickThis post introduces Observatory, a new observability tool that unifies real-user data (RUM), backend telemetry (including TTFB breakdown), error rates (edge and origin), cache hit ratios, and synthetic testing (browser and network tests). It also launches Smart Shield, a feature that applies automated fixes for performance bottlenecks identified by Observatory. The post details the importance of RUM as a source of truth, the diagnostic value of TTFB and its sub-components, and the performance benefits of cache hit ratios. It also describes the new network synthetic test type with histogram-based results.
  24. The RUM Diaries: enabling Web Analytics by defaultThis post announces the upcoming integration of Web Analytics into Cloudflare's performance analytics suite, combining client-side metrics (Core Web Vitals, resource load times, TLS handshake duration) with network-level and origin data. It details the architecture: a lightweight JavaScript beacon in the browser, aggregation at the edge to reduce storage and remove personal information, and processing in core data centers. Web Analytics will be enabled by default for all free domains starting October 15, 2025, with a focus on privacy-first data collection by aggregating data by visits and excluding client-specific information. The post also highlights the ability to exclude EU/UK visitors and the future delivery of a comprehensive, privacy-first platform for performance monitoring and debugging.
  25. A deep dive into Cloudflare’s September 12, 2025 dashboard and API outageThis post details an incident where a bug in the dashboard's React useEffect hook caused repeated, unnecessary calls to the Tenant Service API due to a problematic object in its dependency array. This, combined with a service update to the Tenant Service API, overwhelmed the service and led to a broad outage of APIs and the Cloudflare Dashboard. The response involved increasing resources for the Tenant Service and implementing a temporary rate limit. Going forward, Cloudflare will migrate the Tenant Service to Argo Rollouts for automatic rollback, introduce random delays to dashboard retries to mitigate thundering herd issues, allocate more capacity to the Tenant Service, and enhance API calls with retry/new request indicators for improved observability.
  26. Automating threat analysis and response with CloudyThis post introduces the integration of Cloudflare's AI agent, Cloudy, with its security analytics functionality and a new conversational interface. Cloudy can now assist users with faster root cause analysis (RCA) of traffic anomalies and mitigation by querying security analytics dashboards and logs. Furthermore, Cloudy is integrated with the Cloudforce One Threat Events platform, allowing users to query global threat intelligence data in natural language to understand, prioritize, and act on indicators of compromise (IOCs). The underlying technology leverages the Agents SDK running on Workers AI.
  27. Troubleshooting network connectivity and performance with Cloudflare AIThis post introduces two new AI-powered tools for troubleshooting Cloudflare One: the WARP diagnostic analyzer and an MCP server for Digital Experience Monitoring (DEX). The WARP diagnostic analyzer uses AI to interpret WARP diagnostic logs, providing summaries, event highlights, and detailed explanations to help users diagnose client and network connectivity issues. The DEX MCP server allows users to query device performance data using natural language, simplifying the creation of custom analytics and troubleshooting workflows without requiring extensive data pipeline engineering. The post demonstrates the use of the DEX MCP server to diagnose slow DNS resolution and high resource fetch times for a specific internal application.
  28. Reducing double spend latency from 40 ms to < 1 ms on privacy proxyThis post details the investigation and resolution of high latency (~40 ms) in double-spend checks for Cloudflare's privacy proxy product. The root cause was identified as the interaction between Nagle's algorithm and delayed ACKs in TCP, leading to inefficient packet transmission for small `memcached` commands. The fix involved using `BufWriter<TcpStream>` to buffer writes and send the complete `memcached` command in a single message, reducing latency to <1 ms.
  29. Cloudflare Log Explorer is now GA, providing native observability and forensicsThis post announces the General Availability of Cloudflare Log Explorer. It details the product's capabilities for native observability and forensics, including custom dashboards with AI-assisted chart creation, log search with SQL interface, saved queries, and upcoming custom alerting. The technical implementation leverages Cloudflare R2 for storage and a three-component architecture (ingestors, compactors, queriers) for scalable log processing. It highlights benefits like reduced cost, faster detection, and accelerated investigations.
  30. Cloudflare service outage June 12, 2025This post details a significant outage caused by a failure in the underlying storage infrastructure for Workers KV, which is a critical dependency for many Cloudflare products. The outage impacted services like Access, Gateway, WARP, Dashboard, Challenges, Turnstile, Browser Isolation, Images, Stream, Realtime, Workers AI, Pages, Workers Assets, AutoRAG, Durable Objects, D1, Queues, Event Notifications, AI Gateway, CDN, Workers, Workers for Platforms, Workers Builds, Browser Rendering, and Zaraz. The root cause was a third-party cloud provider outage impacting Workers KV's central data store. The post outlines the specific impacts on each service and the architectural dependencies that led to cascading failures. It highlights that while the proximate cause was a third-party vendor failure, Cloudflare takes ultimate responsibility for its chosen dependencies and architecture.
  31. Performance measurements… and the people who love themThis post details the challenges and techniques involved in measuring and communicating performance metrics, particularly latency. It defines latency from a client's perspective and discusses the limitations of server-side measurements. It also adopts Catchpoint's definition of TTFB (Time to First Byte) as the sum of DNS, Connect, TLS, and Wait times, and illustrates how browser timing metrics align with this definition. The post highlights the importance of understanding statistical concepts like percentiles and the potential for lossy compression to obscure critical data, especially in high-throughput systems. It introduces new tools and techniques for performance analysis and shares surprising results uncovered by these methods.
  32. Scaling with safety: Cloudflare's approach to global service health metrics and software releasesThis post details the implementation of Health Mediated Deployments (HMD) for automated software releases. It explains how HMD uses Prometheus metrics queried via Thanos to monitor Service Level Objectives (SLOs) and automatically revert problematic code. The post highlights the use of R2 for storing historical metric data for backtesting, and details performance optimizations for backtesting, including recording rules and distributed query processing in Thanos. It quantifies the speedup achieved through these optimizations, showing a 3-5x improvement in query response times.
  33. Introducing Workers Observability: logs, metrics, and queries – all in one placeIntroduces Workers Observability, a unified dashboard for querying logs, viewing metrics, and building visualizations across all Workers. Key components include a Query Builder with aggregate functions, filters, search, group by, order by, and limits. It emphasizes structured logs and provides examples of using CPU time and wall time metrics. The post also details a new Invocations View for grouping logs by execution event and announces the Workers Observability API for programmatic access to telemetry data.
  34. Cloudflare incident on March 21, 2025This post details a specific incident where a human error during credential rotation for the R2 Gateway service led to a prolonged outage. The root cause was the inadvertent deployment of new credentials to a development instance instead of production, coupled with a lack of real-time visibility into which credentials were being used by the Gateway Worker. The incident impacted R2, Billing, Cache Reserve, Images, Key Transparency Auditor, Log Delivery, Stream, and Vectorize. The resolution involved identifying the incorrect deployment and deploying the correct credentials to the production environment. The post emphasizes the need for improved visibility into credential management and deployment processes to prevent future authentication failures.
  35. Improving Data Loss Prevention accuracy with AI-powered context analysisIntroduces AI-powered context analysis to Cloudflare's Data Loss Prevention (DLP) solution. This feature leverages Workers AI for text embedding and Vectorize for nearest neighbor search to improve detection accuracy and reduce false positives by learning from customer feedback. It utilizes online clustering to reduce storage and speed up searches, and Cloudflare Queues to separate indexing from the scanning hot path. Privacy is maintained through data redaction and customer-specific namespaces. Limitations include increased latency and English-only support, with plans for multilingual models and broader availability across CASB and Email Security.
  36. Unleashing improved context for threat actor activity with our Cloudforce One threat events platformIntroduced a new threat events platform for Cloudforce One customers, leveraging Cloudflare's network traffic insights to provide contextual data about observed threats. The platform is built using Cloudflare Workers and SQLite-backed Durable Objects for scalable storage of threat events, including IOCs and MITRE ATT&CK/Cyber Kill Chain mappings. It offers access via the Cloudflare Dashboard and a dedicated API, with initial data focusing on DDoS attacks and advanced threat operations, with plans to expand to WAF, Zero Trust Gateway, and Email Security data.
  37. Cloudflare enables native monitoring and forensics with Log Explorer and custom dashboardsThis post announces the expansion of Log Explorer to include logs from Cloudflare's Zero Trust product suite (Access, Gateway DNS, Gateway HTTP, Gateway Network, CASB, Zero Trust Network Session, Device Posture Results). It also introduces custom dashboards that allow users to create visualizations using natural language prompts or templates for monitoring specific metrics and suspicious activity. The integration with Security Analytics for drill-down investigations and rule creation is highlighted.
  38. Automatic Audit Logs: new updates deliver increased transparency and accountabilityThis post introduces Automatic Audit Logs, a new system that unifies and standardizes audit logging across Cloudflare products. Key technical contributions include: a standardized logging format generated automatically via an API Gateway and Worker pipeline, expanded product coverage from 62 to 111 products, granular filtering capabilities, enhanced context (authentication method, API/Dashboard origin, Ray ID mapping), and comprehensive activity capture (including GET requests and failed attempts). The system leverages the Cloudflare OpenAPI Schema for redaction of sensitive information and a data pipeline involving Logfwdr, Logreceiver, and Buftee buffers. An Ingestor service transforms requests into standardized audit log records, and logs are enriched with user/authentication details and stored in a database. The beta release provides 30 days of logs, with GA planned for 18 months of storage and a new dashboard UI.
  39. Cloudflare incident on February 6, 2025This post details a significant incident where human error and insufficient validation safeguards during an abuse remediation action led to the disabling of the R2 Gateway service, impacting R2 object storage and dependent services for 59 minutes. The incident highlights a failure in system-level controls and the need to reduce the blast radius of actions that could disable production services. The post provides a detailed timeline, impact analysis across various services (R2, Stream, Images, Cache Reserve, Log Delivery, Durable Objects, Cache Purge, Vectorize, Key Transparency Auditor, Workers & Pages), and an architectural deep dive into R2's separation of concerns between the Gateway service and its underlying storage and metadata layers. It also discusses the secondary impact on Durable Objects due to client reconnections.
  40. Cloudflare incident on November 14, 2024, resulting in lost logsThis post details a specific incident where a misconfiguration in the Logfwdr configuration system, combined with a latent 'fail open' bug in Logfwdr and an unprovisioned Buftee system, led to the loss of approximately 55% of customer logs for about 3.5 hours. It describes the architecture of the log delivery pipeline (Logfwdr, Logreceiver, Buftee, Logpush) and the cascading failure modes. The root causes identified are the configuration bug and the failure of Buftee to handle a 40x increase in buffer creation due to unconfigured safeguards. Going forward, Cloudflare is implementing new alerts, fixing the bug and associated tests, and planning regular 'overload tests' to simulate similar cascade failures.
  41. Moving Baselime from AWS to Cloudflare: simpler architecture, improved performance, over 80% lower cloud costsBaselime migrated its observability platform from AWS to Cloudflare, rebuilding its data pipeline on Cloudflare Workers, Durable Objects, and Workers Analytics Engine. This resulted in a simplified architecture, improved performance and scalability, and an over 80% reduction in cloud costs. The migration involved replacing AWS Lambda, Kinesis, EC2, and self-hosted ClickHouse with Cloudflare's serverless primitives, leveraging JavaScript-native RPC for inter-component communication and Durable Objects for stateful processing like real-time error tracking and pattern detection. Workers Analytics Engine was used for high-cardinality analytics, with internal changes to support high dimensionality and adaptive bit rate (ABR) for optimized query performance. Cloudflare Queues were adopted for asynchronous messaging, and D1 for transactional data.
  42. Training a million models per day to save customers of all sizes from DDoS attacksThis post introduces an anomaly detection pipeline for DDoS protection that builds statistical models to flag unusual traffic. It details the evolution from naive volumetric models to time series forecasting and finally to a multi-dimensional approach using Principal Component Analysis (PCA) and Mahalanobis distance to identify novel DDoS attacks by analyzing deviations in various traffic features beyond just volume.
  43. What’s new in Cloudflare One: Digital Experience (DEX) monitoring notifications and seamless access to Cloudflare Gateway with China ExpressIntroduces DEX notifications for proactive monitoring within Cloudflare One. These notifications allow administrators to create configurable alert rules based on z-score and SLOs for device connectivity anomalies, test latency (HTTP and Traceroute tests), and test low availability. The post details the problem solved, the statistical methods used (z-score), and provides examples of notification configurations for different scenarios.
  44. Improving platform resilience at Cloudflare through automationThis post details the development of an automatic remediation system built on Temporal to address platform resilience issues at Cloudflare. The system includes a coordinator for authorization, workflow task routing, and safety constraint enforcement using mTLS and ACLs. It leverages Temporal's Visibility API and task routing capabilities to schedule workflows on specific server types or locations. The system is triggered by signal sources like Prometheus and aims to automate remediation of failures before they impact users.
  45. Network trends and natural language: Cloudflare Radar’s new Data Explorer & AI AssistantThis post introduces the Cloudflare Radar Data Explorer, a web-based interface for building complex API queries and visualizing results, and an AI Assistant that translates natural language questions into Radar API calls using Workers AI. It details the technical implementation of the AI Assistant, including the use of prompt engineering and multiple LLM inference calls to handle the complexity of Radar API endpoints, and the visualization capabilities of the Data Explorer.
  46. Removing uncertainty through "what-if" capacity planningThis post introduces 'Scenario Planner', a new system built by the infrastructure team to enable "what-if" capacity planning. It allows modeling hypothetical changes to demand and supply metrics across time and data centers, converting user inputs into a common unit of 'CPU Time' for simulation. The system outputs heatmaps and 'Expected Failovers' views to visualize capacity forecasts and planned server additions, helping to proactively manage infrastructure decisions and avoid customer impact.
  47. Bringing insights into TCP resets and timeouts to Cloudflare RadarThis post introduces a new capability to the Observability Platform by launching a dashboard and API endpoint on Cloudflare Radar that provides near real-time insights into anomalous TCP connections (resets and timeouts) to Cloudflare's network. It details the classification of these anomalous connections, their potential sources (scanners, application shutdowns, network errors, attacks, tampering), and provides guidance on interpreting the data for self-evaluation, exploring new targets, and longitudinal studies. This significantly enhances the platform's ability to diagnose and understand network-level issues.
  48. Introducing HTTP request traffic insights on Cloudflare RadarCloudflare Radar has added a new metric for HTTP request traffic, complementing existing byte-based metrics. This provides a more granular view of traffic patterns, user behavior, and helps distinguish between real events and data pipeline issues. The new metric is available on the Overview and Traffic pages of Radar, and via the Radar API. The post explains the difference between bytes and requests, and how the new metric offers a cleaner signal for user activity, especially for API traffic.
  49. A recent spate of Internet disruptionsThis post details Cloudflare Radar's methodology for detecting and analyzing widespread internet disruptions, focusing on specific incidents in Bangladesh, Syria, and Pakistan. It showcases how Radar uses traffic and IP address space data to identify the scope and timing of outages, including government-directed shutdowns, technical failures, and exam-related disruptions. The post also highlights the impact of these events on network quality metrics and the subsequent restoration of services.
  50. Making WAF ML models go brrr: saving decades of processing timeThis post details performance optimizations for the WAF ML product, specifically focusing on feature extraction. The team replaced HashMap lookups with Aho-Corasick for ngram processing, achieving a 47.84% reduction in processing time for long bodies and a 1.71x speedup for average bodies. Further optimizations are planned for the regex replacement step.
  51. Cloudflare incident on June 20, 2024This post details a specific incident on June 20, 2024, caused by two independent events: suboptimal network traffic re-routing due to automated monitoring and a latent bug in the rate limiting system triggered by a new DDoS mitigation mechanism. The latter event, where a specific HTTP request caused a process to enter an infinite loop, is described in technical detail, including the root cause involving a recursive function call due to an anticipated internal API usage and a broken cookie validation function. The post also outlines the incident timeline, impact metrics (error rates, TTFB latency), and the mitigation steps taken, such as disabling the DDoS rule and restarting services.
  52. How the UEFA Euro 2024 football games are impacting local Internet trafficThis post demonstrates how Cloudflare's global network visibility, a core component of its observability platform, can be leveraged to analyze real-world internet traffic trends during major events like the UEFA Euro 2024. It provides data on traffic drops during games, identifies specific countries impacted, and details observed application-layer DDoS attacks, showcasing the platform's ability to provide insights into network behavior and security incidents during high-profile events.
  53. Exploring the 2024 EU Election: Internet traffic trends and cybersecurity insightsThis post analyzes internet traffic trends and cyberattack activity during the 2024 EU elections. It details observed drops and increases in HTTP requests across various European countries, correlating them with election events and announcements. It also highlights spikes in DNS traffic to election-related sites and news outlets, and discusses DDoS attack activity targeting government-related websites. The analysis uses Cloudflare Radar and DNS data to provide these insights.
  54. Internet insights on 2024 elections in the Netherlands, South Africa, Iceland, India, and MexicoThis post introduces the Cloudflare Radar platform as a tool for tracking and analyzing internet traffic trends during major global events, specifically focusing on national elections in the Netherlands, South Africa, Iceland, India, and Mexico. It details observed patterns of internet traffic decreases during polling hours and increases as results are announced, and highlights instances of cyberattacks, particularly DDoS attacks targeting political websites during the European Parliament election. The post also provides country-specific insights into traffic shifts and attack trends during the elections in Mexico, India, and the Netherlands.
  55. Adopting OpenTelemetry for our logging pipelineThis post details the migration of Cloudflare's logging pipeline from syslog-ng to OpenTelemetry Collector. Key technical contributions include the development of four internal OpenTelemetry Collector components: cfjs1exporter (to support Cloudflare's internal cfjs1 log format), fileexporter (a fork supporting plain text and RFC 5424 syslog formats with custom permissions), externaljsonprocessor (to fetch external data like data center status and add it as fields to logs), and ratelimit processor (to apply rate limits based on log message fields). The migration process involved using OCB to build an internal distribution and phased rollouts across edge and core data centers, with specific salt states and temporary migration pieces. Lessons learned include the need for robust failover mechanisms in the cfjs1 exporter and the importance of handling niche services and 'Internet badness'.
  56. Reclaiming CPU for free with Go's Profile Guided OptimizationThis post details the process of applying Profile Guided Optimization (PGO) to a Go-based service (wshim) within Cloudflare's Observability team. It covers collecting representative CPU profiles from production infrastructure using Thanos and pprof, merging these profiles, and recompiling the service with the `-pgo` flag. The results show a ~3.5% reduction in CPU usage, translating to approximately 97 fewer cores, demonstrating the effectiveness of PGO for optimizing existing code without modification.
  57. Major data center power failure (again): Cloudflare Code Orange testedThis post details Cloudflare's response to a major data center power failure, highlighting the improvements made to control plane resilience and the speed of recovery. Key technical contributions include: - Shifting engineering resources to address high availability issues under 'Code Orange'. - Migrating over 100 databases across 20 clusters to a Highly Available (HA) topology with pre-provisioned capacity to absorb loss. - Enhancing Logpush infrastructure with HA in Portland and an active failover option in Amsterdam, leveraging a massively expanded Kubernetes cluster. - Moving Zero Trust functionality to hundreds of data centers for seamless operation during outages. - Improving Stream product resiliency by handing off compute resources to Amsterdam. - Reducing data center cold start time from 72 hours to 10 hours. - Validating HA capabilities through weekly tests and a February chaos exercise that identified and fixed a flaw in Portland's HA deployment.
  58. Cloudflare acquires Baselime to expand serverless application observability capabilitiesCloudflare has acquired Baselime, an observability platform focused on serverless applications. This acquisition aims to integrate Baselime's capabilities, including high cardinality and dimensionality data handling, OpenTelemetry native support, and an improved developer experience with AI-assisted debugging, into the Cloudflare ecosystem. The goal is to provide built-in observability for Cloudflare's serverless services like Workers, Queues, KV, Durable Objects, and AI. Baselime's existing products will be made free and eventually sunsetted into the Cloudflare dashboard by the end of 2024. Future plans include deeper integration with codebases, enhanced AI capabilities for debugging and PR generation, and dedicated observability for LLM applications.
  59. Minimizing on-call burnout through alerts observabilityThis post details the enhancement of alert observability by aggregating all alert states (firing, silenced, inhibited, and resolved) into a datastore using Vector.dev and ClickHouse. It describes the lifecycle of an alert within Prometheus and Alertmanager, the limitations of the previous alert reporting mechanism (alertmanager2es), and the proposed solution involving correlating data from Alertmanager webhooks and API endpoints using the fingerprint field. The implementation uses Vector.dev for data transformation and ClickHouse for storage, with a flexible schema to accommodate various alert labels. The outcome is multiple dashboards for comprehensive alert analysis.
  60. Introducing Requests for Information (RFIs) and Priority Intelligence Requirements (PIRs) for threat intelligence teamsCloudforce One built an internal threat investigation platform using Cloudflare Workers, Pages, R2, KV, D1, and Durable Objects. This platform manages RFIs, PIRs, threat cases, leads, and a binary database. The case management service uses Durable Objects with WebSockets for real-time collaboration and persistent storage. Vectorize is used for searching case files, with content indexed and embedded into vectors. Cloudflare Queues are used to push data to an indexing engine consumer.
  61. Cloudflare’s URL Scanner, new features, and the story of how we built itThis post introduces the URL Scanner, a new tool within Cloudflare Radar that detects and safeguards against malicious websites. It details the integration of the URL Scanner with the Security Center in the Cloudflare Dashboard, enhancing cybersecurity workflows. The post also highlights the new URL Scanner API, enabling custom scans with features like custom HTTP headers and multi-device screenshots, and explains the underlying technology stack, emphasizing the use of Cloudflare Workers, Durable Objects, and the Browser Rendering API.
  62. Log Explorer: monitor security events without third-party storageIntroduces Log Explorer, a new feature for native storage and querying of HTTP Request and Security Event logs within the Cloudflare Dashboard. Log Explorer is built on Cloudflare R2 and the Delta Lake protocol, offering ACID compliance and a SQL interface. It allows for granular investigation of security events, false positives, and specific requests via Ray ID, reducing reliance on third-party storage and analysis tools. The implementation uses Rust and open-source libraries like delta-rs and Apache Arrow DataFusion.
  63. Free network flow monitoring for all enterprise customersIntroduces a free version of Magic Network Monitoring for all Enterprise Customers, enabling them to send NetFlow or sFlow data to Cloudflare for network traffic analysis. Details how this product was used internally during the Thanksgiving 2023 security incident and outlines use cases for monitoring local IoT and cloud network traffic. Provides instructions for Enterprise Customers to gain access and mentions future support for native cloud VPC flow logs.
  64. Cloudflare launches AI Assistant for Security AnalyticsIntroduces an AI Assistant for Security Analytics that allows users to query security event data using natural language. The assistant leverages Workers AI and LLMs to convert natural language queries into GraphQL filters for Cloudflare's analytics APIs, generating time series charts for comparisons. This removes the complexity of manual filtering and enables unconstrained data interrogation, keeping customer data private and within Cloudflare's network.
  65. Enhancing security analysis with Cloudflare Zero Trust logs and Elastic SIEMThis post details a new integration between Cloudflare Zero Trust logs and Elastic SIEM. It introduces pre-built dashboards for analyzing Zero Trust logs (Gateway HTTP, CASB, Access Requests, Audit logs, DNS, Network, HTTP, Zero Trust Session Logs) within Elastic, enabling enhanced visibility, field normalization, efficient search, and correlation for threat detection. The integration supports Logpush jobs via HTTP Endpoint, AWS S3 polling, or SQS modes, and guides users on enabling the Cloudflare Logpush integration within Kibana.
  66. Monitoring machine learning models for bot detectionThis post details the implementation of a machine learning monitoring system for Cloudflare's Bot Management product. It describes the challenges of monitoring ML models in production, the process of generating ground-truth datasets, the use of specializations for granular analysis, and the integration of this monitoring system into the Endeavor MLOps platform. The system leverages SQL queries against ClickHouse for data fetching, a MetricsComputer for calculating performance metrics, and runs on GPUs for efficient computation.
  67. Fulfilling the promise of single-vendor SASE through network modernizationThis post announces updates to Cloudflare One, the SASE platform, focusing on network modernization to fulfill the promise of single-vendor SASE. It introduces more flexible on-ramps for site-to-site connectivity (agent/proxy-based and appliance/routing-based), WAN-as-a-service (WANaaS) capabilities (high availability, application awareness, VM deployment, enhanced visibility/analytics), and Zero Trust connectivity for DevOps (mesh and peer-to-peer secure networking). This enhances the platform's ability to provide unified network and security visibility and control for various teams.
  68. Thanksgiving 2023 security incidentThis post details a security incident where a nation-state actor gained access to Cloudflare's internal Atlassian server using compromised credentials from a previous Okta breach. The actor accessed documentation and source code related to network architecture and management. Cloudflare's response involved a 'Code Red' remediation effort, including rotating over 5,000 production credentials, reimaging every machine in the global network, and performing forensic triages. The incident highlights the importance of robust credential management, zero-trust architecture, and the role of observability in detecting and responding to sophisticated attacks. While the incident did not impact customer data or systems, it led to significant internal hardening efforts and a deeper investigation into access controls and system integrity.
  69. Introducing Foundations - our open source Rust service foundation libraryIntroduced Foundations, an open-source Rust library for production-grade distributed systems. Foundations provides a unified API for observability, encompassing logging, tracing, and metrics. Key features include simplified telemetry setup, enhanced trace sampling flexibility, distributed trace stitching, trace forking for long-lived connections, implicit context propagation for logs, and a streamlined API for defining Prometheus metrics with typed labels.
  70. An overview of Cloudflare's logging pipelineThis post details the evolution of Cloudflare's internal logging pipeline, which is a critical component of the Observability Platform. It covers the journey from basic logging to a sophisticated, highly available system capable of shipping close to a million log lines per second. Key contributions include: - Describing the use of various logging libraries (zerolog, KJ_LOG, log) and their integration with systemd-journald. - Explaining the role of syslog-ng for log collection, rate limiting, field rewriting, and JSON wrapping. - Detailing the 'log-x' operation for achieving redundancy across two core data centers (log-a, log-b) and using Kafka Mirror Maker with a Dead Letter Queue (DLQ) to handle transient data center disconnections. - Explaining the use of Kafka for buffering logs, its benefits for consumer flexibility and fault tolerance, and the partitioning strategy using host and service name to guarantee ordering within a partition. - Describing the long-term storage backends: an Elasticsearch/Logstash/Kibana (ELK) stack with dedicated master, data, and HTTP nodes, and a Clickhouse cluster for service logs. - Outlining future projects including multi-tenancy, migration to Open Telemetry Logs, tail sampling, and Kafka cluster balancing.
  71. Better debugging for Cloudflare Workers, now with breakpointsThis post introduces and details the implementation of breakpoint debugging for Cloudflare Workers using the Chrome DevTools Protocol (CDP). It explains how the workerd runtime was modified to include an I/O thread to handle CDP messages, allowing JavaScript execution to be suspended at breakpoints without blocking the event loop. The post also covers the use of source maps for mapping generated JavaScript back to original source code and provides setup instructions for VSCode and WebStorm.
  72. Post mortem on the Cloudflare Control Plane and Analytics OutageThis post details a major outage of Cloudflare's control plane and analytics services caused by a power failure at a third-party data center. It highlights the failure of a critical data center provider, the unexpected behavior of backup power systems, and the lack of communication from the provider. Crucially, it reveals that while the core network remained operational, critical control plane and analytics systems (Kafka, ClickHouse) were not fully integrated into the high availability cluster and had non-obvious dependencies on the failed data center, leading to extended downtime. The post outlines intended design for data center failure, what failed (dependencies on PDX-04, incomplete HA cluster migration, lax GA requirements for new products), what worked (core network, disaster recovery facility for some services), and proposed changes to ensure all critical systems are in the HA cluster and dependencies are better managed.
  73. Cloudflare incident on October 30, 2023This post details a significant incident where a misconfiguration in a new deployment tool for Workers KV led to a 37-minute outage of multiple Cloudflare services. The incident highlights the critical dependency on Workers KV for various products (Pages, Access, WARP, AI Gateway, etc.) and the challenges of resolving issues when core internal tooling itself is impacted. Key takeaways include the need for improved deployment tooling safety, better rollback mechanisms that are resilient to outages of dependent services (like Access), and reducing the blast radius of critical dependencies.
  74. Introducing notifications for HTTP Traffic AnomaliesIntroduced Traffic Anomalies notifications for enterprise customers, which trigger based on unexpected changes in HTTP traffic volume. This feature utilizes z-scores to compare short-term (5-minute) traffic averages against long-term (4-hour) averages, alerting when deviations exceed 3.5 standard deviations. Minimum thresholds of 200 requests for spikes and drops are applied to reduce noise. The system stores sampled statistics in Clickhouse, processes them in VictoriaMetrics, and leverages the existing Alert Notification System for delivery.
  75. Network flow monitoring is GA, providing end-to-end traffic visibilityThis post announces the general availability of Magic Network Monitoring (MNM), a cloud network flow monitor. It details the technical challenges and solutions involved in improving traffic volume estimations in MNM, specifically the transition from simple random sampling to VarOpt reservoir sampling to accurately handle the long-tail distribution of network flow data weights. It also outlines how MNM integrates with Magic Transit and Magic WAN for enhanced DDoS detection and network visibility.
  76. Cloudflare Integrations Marketplace introduces three new partners: Sentry, Momento and TursoIntroduces three new integrations for Cloudflare Workers: Sentry for error monitoring and logging, Turso (based on libSQL/SQLite) for edge-hosted distributed databases, and Momento Cache for low-latency serverless caching. The Sentry integration uses a Tail Worker to capture and send errors, exceptions, and console logs. Turso integration automatically pulls API credentials and uses the libsql SDK. Momento integration uses OAuth2 for API key retrieval and the Momento SDK. All integrations aim to reduce developer busywork and glue code.
  77. Traffic anomalies and notifications with Cloudflare RadarThis post introduces the integration of an internal traffic anomaly detection and alerting tool into Cloudflare Radar's Outage Center (CROC). It also launches new Radar notification functionality, allowing users to subscribe to alerts for traffic anomalies, confirmed Internet outages, route leaks, or route hijacks at country or autonomous system levels. The post details the technical implementation of displaying traffic anomalies on the CROC map and timeline, providing API access to traffic anomaly data, and configuring notifications via the Cloudflare dashboard.
  78. Gone offline: how Cloudflare Radar detects Internet outagesCloudflare Radar is enhancing its outage detection capabilities by publishing anomalous traffic events for countries and Autonomous Systems (ASes). This involves automatically detecting traffic drops, verifying them with internal and external data sources, and allowing users to subscribe to notifications. The post details the time-series analysis approach, including decomposition into seasonal, trend, and residual components, and discusses various traffic patterns observed in different regions and ASes. It also outlines the preprocessing steps required for effective time-series modeling and the challenges of real-time anomaly detection.
  79. Announcing Cloudflare Incident AlertsThis post introduces Cloudflare Incident Alerts, a new feature that provides proactive, customizable notifications for incidents affecting customer services. Alerts can be delivered via email, webhook, or PagerDuty and are configurable by product and impact level, offering a push-based communication model to supplement the existing Status API and RSS feeds. This aims to reduce confusion and wasted resources during incidents by providing timely and relevant information directly to users.
  80. Traffic transparency: unleashing the power of Cloudflare TraceIntroduced Cloudflare Trace, a new product that provides detailed visibility into how HTTP requests traverse Cloudflare's configuration and which rules are applied. This feature allows users to understand the journey of their traffic through Cloudflare's complex ecosystem, debug issues related to rule application, and export trace data for support tickets. The development of Trace was informed by internal tools like Flute and customer feedback, aiming to strike a balance between detail and usability.
  81. Cloudflare Email Security now works with CrowdStrike Falcon LogScaleThis post details the integration of Cloudflare Email Security detection data with CrowdStrike Falcon LogScale. It provides step-by-step instructions for setting up the webhook integration, including generating an ingest token and configuring the target URL. The post also highlights the out-of-the-box dashboard and parser provided within Falcon LogScale for visualizing and querying this email security data. Future plans include extending this integration to other components of the Zero Trust Suite.
  82. How Waiting Room makes queueing decisions on Cloudflare's highly distributed networkThis post details the evolution of Cloudflare's Waiting Room product, focusing on its core mechanism for queueing traffic during spikes. It explains how Waiting Room leverages Cloudflare Workers and Durable Objects for distributed decision-making to minimize latency. The post elaborates on the configuration options (total active users, new users per minute, session duration) and the algorithm used at the worker level to decide when to queue users, considering traffic history and distribution across data centers and workers to optimize user experience and prevent origin overload.
  83. Improving Worker Tail scalabilityImproved Workers Tail scalability by moving log filtering out of Durable Objects to the message producer, reducing load on the Durable Object. Introduced a sampling mechanism to keep Durable Object RPS below 100 when filters are insufficient. Added a failsafe to stop forwarding messages if the Durable Object becomes overloaded.
  84. Cloudflare Radar’s 2023 overview of new tools and insightsThis post details the introduction of several new features to Cloudflare Radar in 2023: a URL Scanner for comprehensive website analysis (risk assessment, SSL, HTTP data, performance, DNS, technologies), an Internet Quality page providing country and AS-level insights on bandwidth, latency, and jitter, Trending Domains to highlight daily and weekly surges in domain popularity, and a Routing page that examines BGP status, secure routing protocol deployment, route leaks, and hijack detection. It also recaps previously published insights on Internet trends during major global events like the war in Ukraine, Virgin Media outage, holidays, and conflicts in Sudan and Niger.
  85. Integrate Cloudflare Zero Trust with Datadog Cloud SIEMThis post announces the general availability of the Cloudflare Zero Trust Integration with Datadog, which includes a Cloudflare Content Pack within Datadog Cloud SIEM. This integration provides out-of-the-box dashboards and detection rules for customers ingesting Zero Trust logs into Datadog, offering improved security insights. Specific features include enhanced security insights over the Zero Trust landscape, security detections built for CASB findings, and security detections for impossible travel scenarios. The integration facilitates log ingestion via Logpush jobs and installation of the Cloudflare Tile in Datadog.
  86. Measuring the Internet's pulse: trending domains now on Cloudflare RadarThis post introduces new 'Trending Domains' lists to Cloudflare Radar, complementing the existing Domain Rankings. The 'Trending Today' list captures sudden spikes in popularity by comparing a domain's current rank to its best rank over the previous four days, quantifying the increase as a percentage change. The 'Trending This Week' list highlights domains with steady growth by analyzing the weighted average daily rank improvement compared to the best rank of the previous six days, with more recent days weighted more heavily. The methodology for selecting the baseline list size for trending analysis is also detailed, varying by location based on DNS query distribution.
  87. The day my ping took countermeasuresThis post details the evolution of the `ping` utility, a fundamental network diagnostic tool, highlighting its 'countermeasures' against clock manipulation. It explores how `ping` handles time synchronization issues, the underlying mechanisms like `vdso` and `CAP_NET_RAW` capabilities, and the use of `strace` for debugging. This contributes to the broader observability platform by demonstrating how even basic network tools are being refined for greater accuracy and reliability in performance measurement.
  88. Recapping Speed Week 2023This post recaps Speed Week 2023, highlighting new product launches and updates related to performance measurement and optimization. Key contributions include the introduction of Observatory, Digital Experience Monitoring, and Timing Insights for application and website performance analysis. It also details network performance updates, the fastest Secure Web Gateway, ZTNA, and Remote Browser Isolation solutions, and network growth. Several product announcements are made to simplify speed boosts, including Smart Hints, HTTP/3 Prioritization, Argo for UDP, Brotli end-to-end, LL-HLS for Stream, and Ricochet for API Gateway. The post also touches on AI/ML advancements for performance-sensitive inference and scaling machine learning at the microsecond level, and mentions migration programs like Turpentine and Descaler.
  89. Introducing the Cloudflare Radar Internet Quality PageIntroduced the Cloudflare Radar Internet Quality page, which provides country and network (autonomous system) level insights into Internet connection performance (bandwidth) and quality (latency, jitter) over time. This page aggregates data from Cloudflare's speed test tool and other measurement targets to offer a comprehensive view of internet connectivity characteristics.
  90. Understanding end user-connectivity and performance with Digital Experience Monitoring, now available in betaIntroduced Digital Experience Monitoring (DEX) as a beta feature for Cloudflare One customers. DEX provides Synthetic Application Monitoring to measure performance and availability of public or private applications from various global locations, visualizing resource fetch times, server and DNS response times, and HTTP status codes. Additionally, Fleet Status provides real-time insights into the status, connection mode, and location of WARP-enrolled devices, offering global snapshots, per-device details, and time-series charts for administrators.
  91. Introducing Timing Insights: new performance metrics via our GraphQL APIIntroduces 'Timing Insights' via the GraphQL Analytics API, providing new metrics to break down Time to First Byte (TTFB). New fields include `edgeTimeToFirstByteMs`, `edgeDnsResponseTimeMs`, and `originResponseDurationMs`, available at various quantiles (P50, P95, P99). Demonstrates how to use these metrics to diagnose performance bottlenecks, specifically highlighting DNS resolution as a potential cause for slow TTFB.
  92. Exam-related Internet shutdowns in Iraq and Algeria put connectivity to the testThis post details the use of Cloudflare Radar to track and analyze government-imposed internet shutdowns in Iraq and Algeria during academic exams. It showcases how Radar's network-level traffic data, BGP announcement activity, and 1.1.1.1 DNS resolver traffic can be used to identify and characterize these disruptions, distinguishing between routing-based shutdowns and content blocking. The post also highlights a new Cloudflare Radar feature for embedding interactive graphs directly into blog posts, enabling direct visualization of this data.
  93. Cloudflare’s view of Internet disruptions in PakistanThis post details the impact of government-directed mobile internet shutdowns and social media platform blocking in Pakistan on internet traffic patterns. It analyzes national and local traffic trends, shifts in mobile vs. desktop usage, changes in human vs. bot traffic, and the increased usage of Cloudflare's 1.1.1.1 resolver. It also examines network-level outages across major mobile providers and the subsequent increase in traffic on fixed broadband networks, as well as observed latency changes. The analysis extends to administrative unit and city-level disruptions and the impact of content blocking on specific social media platforms.
  94. How we built Network Analytics v2This post details the redesign of Cloudflare's Network Analytics system (v2), which provides real-time visibility into network layer traffic patterns for Magic Transit and Spectrum customers. The redesign addresses limitations of the previous version by enabling the system to ingest and analyze packet samples with context-specific metadata from various network functions (e.g., DDoS mitigation, firewalling, load balancing), inspired by traditional network monitoring tools like Netflow and sFlow. Key technical challenges and solutions are discussed, including the design of the 'samplerd' data pipeline and modifications to the xdpd daemon to emit metadata-rich packet samples, even for dropped packets, while optimizing for performance.
  95. Effects of the conflict in Sudan on Internet patternsThis post details Cloudflare's observation and analysis of internet traffic disruptions in Sudan due to armed conflict. It presents data on traffic drops, ISP outages, BGP update spikes, and shifts in mobile traffic percentage and messaging app usage. The post uses Cloudflare Radar data to illustrate the impact of the conflict on internet connectivity and user behavior, highlighting specific ISPs and timeframes of disruption.
  96. Measuring network quality to better understand the end-user experienceThis post introduces the Aggregated Internet Measurement (AIM) initiative, a new open format for displaying internet quality that aims to bridge the gap between traditional speed test metrics (bandwidth, packet loss, latency, jitter) and the end-user experience. AIM provides use-case specific performance insights while retaining detailed network data for engineers. The post details the challenges of current speed tests, explains the meaning of core network metrics, and illustrates how AIM can help diagnose issues at the local network (WiFi) or broader internet level. It also announces the partnership with Measurement Lab and the open-sourcing of the speedtest code and AIM score calculations.
  97. Introducing Cloudflare’s new Network Analytics dashboardIntroduced a new Network Analytics dashboard for Magic Transit and Spectrum customers, providing enhanced visibility into network traffic, firewall events, and DDoS attacks. This dashboard is powered by a refactored, decentralized data logging pipeline that offers improved resilience, performance, and scalability. Key features include detailed traffic attribute analysis, DDoS mitigation insights, Magic Firewall event visibility, and packet sampling. The dashboard enables granular drill-downs by various dimensions, accurate geographical information based on data center ingestion, and detailed mitigation analytics. It facilitates faster DDoS response by allowing pattern identification and the creation of Magic Firewall rules directly from the interface, and aids in generating security reports.
  98. Internet disruptions overview for Q1 2023This post details observed Internet disruptions in Q1 2023, categorizing them by cause (government-directed shutdowns, cable cuts, power outages) and providing specific examples with associated network traffic data and affected providers. It highlights the impact of these events on global Internet resilience and provides links to further data on Cloudflare Radar.
  99. Analyze any URL safely using the Cloudflare Radar URL ScannerIntroduced the Radar URL Scanner, a free tool that compiles detailed technical reports for any given URL. The scanner provides data across Security, Cookies, Network, Technology, DOM, and Performance categories. It leverages Cloudflare's Workers Browser Rendering API for headless scans and offers insights into phishing, SSL certificates, HTTP transactions, DNS records, cookie attributes, technologies used, DOM elements, and performance metrics. Future plans include API endpoints, private scans, security recommendations, and aggregate trend visualizations.
  100. Adding Zero Trust signals to Sumo Logic for better security insightsThis post details the integration of Cloudflare's Zero Trust logs (from Gateway, Remote Browser Isolation, Data Loss Prevention, Access, and Cloud Access Security Broker) with Sumo Logic's Cloud SIEM. It describes how this integration enables automated normalization and correlation of security signals, reducing alert fatigue and accelerating security analyst triage by converging security and network data into high-fidelity insights. It also mentions the existing Cloudflare App for Sumo Logic and outlines the steps for customers to enable Logpush to Sumo Logic and install the Cloudflare App for enhanced Zero Trust visibility.
  101. How Cloudflare runs Prometheus at scaleThis post details Cloudflare's large-scale Prometheus deployment, managing 916 instances and ~4.9 billion time series. It explains Prometheus's data model (metrics, samples, time series), the concept of cardinality, and the problems associated with high cardinality, particularly memory consumption. The post outlines the operational flow of Prometheus, from HTTP scraping to TSDB storage, and discusses the resource implications of each time series. It emphasizes the importance of understanding label usage to prevent cardinality explosions.
  102. Get notified about the most relevant events with Advanced HTTP AlertsThis post introduces advanced HTTP alerts for Enterprise customers, allowing for more granular filtering and organization of alerts based on origin response status codes, edge response status codes, alert sensitivity/SLO, client IPv4/IPv6 addresses, and specific zones. This enhances the observability platform by exposing more request metadata for monitoring.
  103. Introducing Digital Experience MonitoringIntroduces Digital Experience Monitoring (DEM) as a new product to provide visibility into end-user connectivity and performance. DEM will offer features like Zero Trust Fleet Status (monitoring device agent deployment), Synthetic Application Monitoring (creating synthetic tests for public endpoints with traceroute and HTTP GET requests), and Network Path Visualization (hop-by-hop telemetry data). These features aim to help administrators pinpoint and resolve issues impacting user productivity and application availability.
  104. Cloudflare Radar 2022 Year in ReviewThis post details Cloudflare's 2022 Year in Review, focusing on aggregate data analysis of Internet trends. It presents observations on global Internet traffic growth (23% in 2022), traffic trends influenced by events like the Olympics and FIFA World Cup, and the impact of geopolitical events (Ukraine conflict) and infrastructure issues (Puerto Rico power outages) on connectivity. It also breaks down traffic by popular content categories globally and in specific regions like South Korea, Turkey, and Armenia. While it discusses data analysis and trends, it does not introduce new technical systems or significant architectural changes to the observability platform itself, but rather uses the platform's data to report on broader internet trends.
  105. Partnering with civil society to track Internet shutdowns with Radar Alerts and APIThis post details the development and release of Radar Internet shutdown alerts and the Radar Outage Center, alongside the expansion of Radar API access. These tools leverage Cloudflare's network data to provide civil society organizations with insights into internet disruptions, aiding in their documentation and advocacy efforts. The post highlights partnerships with organizations like Access Now, Internet Society, and OONI, showcasing how they utilize these tools.
  106. Send Cloudflare Workers logs to a destination of your choice with Workers Trace Events LogpushIntroduced Workers Trace Events Logpush, a feature that allows developers to export logs from Cloudflare Workers to external destinations like object storage or analytics platforms. This feature provides detailed metadata about requests, console logs, and exceptions, enhancing production observability beyond real-time streaming. The post details configuration via the API and wrangler.toml, sample log formats, and pricing.
  107. How Cloudflare instruments services using Workers Analytics EngineThis post details how Cloudflare uses its own Workers Analytics Engine to instrument its SQL API. It describes logging events for every query to the SQL API, including query performance, errors, and customer query patterns. This data is used for technical observability (request counts, error rates, response times) and product insights (prioritizing features based on customer errors and usage). It highlights the use of Adaptive Bit Rate (ABR) for handling trillions of data points and the integration with Grafana for visualization. Specific examples include monitoring query response times and analyzing customer-encountered SQL errors to inform feature prioritization.
  108. Indexing millions of HTTP requests using Durable ObjectsThis post details the implementation of the R2 Log Retrieval API using Cloudflare Workers, Durable Objects, and the Streams API. It explains how Durable Objects are used to maintain forward indexes of RayIDs for logs stored in R2, enabling efficient lookup without managing a traditional database. The Streams API is utilized to handle large log batches and avoid out-of-memory errors. The post also describes how RayID and batch naming conventions are used to optimize search queries by reducing the number of batches to check.
  109. Store and process your Cloudflare Logs... with CloudflareIntroduces Cloudflare Logs Engine, a new product for storing and analyzing Cloudflare logs within Cloudflare. Initially, it allows Enterprise customers to store logs in R2 and retrieve them by time range. Beta access for RayID retrieval is available. Future plans include support for all plan types, arbitrary filtering, cross-dataset queries, and integration with external data sources. The product aims to address cost and usability challenges of third-party log analysis tools.
  110. Don't roll your own high cardinality analytics, use Workers Analytics EngineIntroduces Workers Analytics Engine, a new service for storing and analyzing high-cardinality time-series data from Cloudflare Workers. It is used internally to monitor the Instant Logs backend and provides a SQL interface for querying. Key improvements for the open beta include enhanced error messaging, a comprehensive SQL reference, and the exposure of the `_sample_interval` column for handling varying data resolutions. Future plans include alerting, named fields, and an integrated dashboard UX.
  111. The status page the Internet needs: Cloudflare Radar Outage CenterThis post introduces the Cloudflare Radar Outage Center (CROC) as an archive for information on observed Internet outages. It details the information contained within CROC (location, ASN, type, scope, cause, start/end time) and how it can be used via the Radar interface and API. The post also outlines future plans for increased automation of outage detection and the extension of CROC to include status pages for cloud service providers.
  112. Monitor your own network with free network flow analytics from CloudflareThis post introduces a free version of Magic Network Monitoring (MNM), previously Flow Based Monitoring, which provides network flow analytics. MNM allows customers to collect network analytics without installing physical devices by configuring routers to send flow data to Cloudflare. The analytics dashboard offers filtering by protocols, source/destination IPs, ports, TCP flags, and router IP. It also includes network traffic volume alerts per IP address or prefix. The post details use cases for monitoring home labs and identifying network layer DDoS attacks, and mentions integration with Magic Transit for DDoS mitigation.
  113. Logpush: now lower cost and with more visibilityThis post introduces three key enhancements to Cloudflare's Logpush product: 1. Log filtering before pushing to customer destinations, allowing users to select specific log criteria (e.g., status codes, firewall actions) to reduce data volume and cost. 2. Alerting for failing Logpush jobs, notifying users via email if a job has been retrying and failing for 24 hours. 3. A GraphQL API for querying Logpush job health statistics, including bytes pushed, records pushed, and push status, providing greater visibility into job performance.
  114. Store and retrieve your logs on R2This post introduces the capability to store and retrieve Cloudflare logs directly on Cloudflare R2 object storage. It details how to configure Logpush jobs to send logs to R2 using either the dashboard or API, and how to retrieve logs using the Log Retrieval API. The post highlights the cost-effectiveness of R2 due to no egress fees and its S3 API compatibility, enabling existing tooling to be used. It also mentions future plans for log analysis and forensics capabilities on R2.
  115. Log analytics using ClickHouseThis post details the migration of Cloudflare's error logging pipeline from Elasticsearch to ClickHouse. It outlines the challenges faced with Elasticsearch, including mapping explosion, poor multi-tenancy support, operational overhead, and garbage collection issues, especially at Cloudflare's scale of 35-45 million HTTP requests per second with 500K-800K failures per second. The solution leverages ClickHouse's column-oriented storage, sparse indexing, and efficient compression (LZ4, Double-Delta, Gorilla, LowCardinality) to achieve faster query performance and cost-effective storage for all error logs. The post also describes the optimized inserter using Cap'n Proto and scalable pods, and discusses ClickHouse's linear scalability for reads and writes.
  116. How Cloudflare One solves your observability problemsThis post announces Cloudflare One Observability, a new capability that unifies data from various Cloudflare One applications (SASE platform) to provide end-to-end visibility for troubleshooting network connectivity, security policies, and performance issues. It highlights how this unified approach addresses the fragmentation of modern enterprise networks, contrasting it with traditional 'castle-and-moat' models and the disorganized combination of tools required today. The post provides customer examples of troubleshooting bandwidth issues and identifying network vulnerabilities, emphasizing the single-pass inspection architecture of Cloudflare One for seamless analysis of traffic and identification of bottlenecks. It also mentions leveraging existing data pipelines like Instant Logs and ABR.
  117. Monitoring our monitoring: how we validate our Prometheus alert rulesIntroduced `prom-rule-validator`, an open-source tool to validate Prometheus alert rules. The tool checks for syntactical correctness and simulates rule execution against a local Prometheus instance to catch potential issues like typos, stale metrics, or incorrect range query usage that could lead to silent failures in alerting. This addresses the challenge of ensuring alert reliability by providing a mechanism to proactively test alert rules before deployment.
  118. Integrating Network Analytics Logs with your SIEM dashboardIntroduces Network Analytics Logs for Magic Transit, Magic Firewall, Magic WAN, and Spectrum, allowing Enterprise customers to push packet samples to SIEM systems like Splunk. Details the log format, available fields, and provides a step-by-step guide for setting up a Logpush job to send these logs to Splunk via API.
  119. Introducing Workers Analytics EngineIntroduces Workers Analytics Engine, a new time-series analytics service built on Cloudflare Workers. It allows users to collect telemetry data from Workers, enabling use cases like application monitoring (e.g., R2 storage), custom security rules (e.g., fail2ban implementation), and tracking external data (e.g., IoT sensors). The engine uses a structured event log model with blobs (strings for filtering/grouping) and doubles (numbers for aggregation), offering advantages over traditional metrics systems like unlimited cardinality, low-latency reporting, and fast queries across timescales. It provides SQL and GraphQL APIs for data querying and integrates with Grafana for visualization.
  120. Logs on R2: slash your logging costsThis post announces the integration of Cloudflare's Logpush service with R2 object storage for enterprise customers. This enables storing raw logs (HTTP, WAF, Spectrum) on R2 at a significantly lower cost compared to S3, offering up to 86% savings. It also introduces plans for a Log Retrieval API to access data stored on R2, starting with time period and rayId lookups, and eventually more complex queries. The post details the setup process for Enterprise customers and previews upcoming Logpush features like new datasets, log filtering, and custom log formatting.
  121. Workers visibility: announcing Logpush for Worker’s Trace EventsThis post announces the upcoming integration of Workers execution logs into Logpush for Enterprise customers. It details the structure of 'Trace Events,' which include unstructured console.log() messages, exceptions, and metadata about requests/responses. This new dataset will allow Enterprise customers to send Workers logs to their preferred cloud storage or analysis platforms, enabling detailed debugging, performance analysis, and enhanced visibility for SaaS companies building on Workers.
  122. PIPEFAIL: How a missing shell option slowed Cloudflare downThis post details a specific incident where a missing 'pipefail' shell option in a Kubernetes cron job led to corrupted configuration data in Quicksilver. This corruption caused the 'dosd' service to panic, which in turn caused a slowdown in the Front Line service due to cache misses and blocking requests while waiting for 'dosd'. The fix involved enabling 'pipefail' in the Bash script and manually re-running the cron job. The incident highlights the importance of robust error handling in shell scripts and the cascading effects of seemingly small configuration errors on critical systems.
  123. Cloudflare ObservabilityThis post outlines the future vision for Cloudflare observability, detailing its three core components: monitoring, analytics, and forensics. It discusses the evolution of notifications, the enhancement of analytics dashboards for security and general traffic analysis, and the expansion of log access through Logpull, Logpush, and Instant Logs, with plans for log storage on R2. Tracing capabilities are also highlighted as a means to understand end-to-end request lifecycles and identify bottlenecks within services like Workers.
  124. Packet captures at the edgeIntroduced on-demand packet captures from Cloudflare's global network for customers using Magic Transit and Magic WAN. This feature allows users to capture packet data via an API, filtering by IP address, ports, and protocol. The implementation uses nftables for filtering and nflog for logging packets, with tcpdump used in user space to generate pcap files. Packet capture files are delivered directly to customer-chosen cloud storage, ensuring data privacy.
  125. Get full observability into your Cloudflare logs with New RelicThis post details the integration of Cloudflare's Logpush service with New Relic, enabling customers to send Cloudflare HTTP request logs directly to New Relic for analysis. It highlights the benefits of this direct integration, including faster log delivery, reduced costs by eliminating middleware, and the ability to correlate Cloudflare data with other telemetry in New Relic One for end-to-end visibility. A quickstart guide and setup instructions are provided for users to implement this integration.
  126. Slicing and Dicing Instant Logs: Real-time Insights on the Command LineThis post introduces the ability for Enterprise customers to access 'Instant Logs' via a WebSocket endpoint. It details how to create a new log session using the Cloudflare API, specifying desired fields (e.g., ClientIP, FirewallMatchesActions) and filters (e.g., ClientRequestPath). It also demonstrates how to use external tools like Websocat to connect to the WebSocket and Angle Grinder to process the streamed JSON logs for analysis, such as counting firewall actions by country. This enables real-time debugging and validation of security rules directly from the command line.
  127. What’s new with Notifications?This post details the evolution of Cloudflare's Notifications platform, introducing webhook support for direct integration with various messaging and monitoring systems (Slack, Teams, Discord, DataDog, Splunk, OpsGenie) and the addition of notification history for the past 30-90 days, accessible via API. This enhances the observability of system events by providing more direct and persistent channels for alerts.
  128. Profiling Your Workers with WranglerIntroduces beta support for local CPU profiling of Cloudflare Workers using Wrangler and integration with Chrome DevTools protocol. This allows developers to inspect and analyze the performance of their Workers by gathering stack traces at a sampled rate, providing insights into CPU-intensive functions and potential bottlenecks. The feature leverages `wrangler dev --inspect` and the `chrome://inspect` interface to visualize flame graphs and annotated source code.
  129. Unboxing the Last Mile: Introducing Last Mile InsightsIntroduces Last Mile Insights, a new tool that leverages Network Error Logging (NEL) and machine learning to provide customers with visibility into connection failures occurring between end-users and Cloudflare's edge. This feature aims to help customers diagnose and understand issues originating in the 'Last Mile' of network connectivity, which is typically managed by ISPs.
  130. How we built Instant LogsIntroduced 'Instant Logs' for real-time HTTP log streaming via WebSockets, complementing the existing Logpush product. Designed a new edge-native pipeline using Workers and Durable Objects to achieve sub-three-second latency. Implemented MapReduce-like filtering with Workers and sharding with Durable Objects to handle varying traffic volumes. Utilized Reservoir Sampling for controlled data reduction and statistical preservation. The system architecture involves web servers sending logs via HTTP POST to Filter Workers, which then route filtered messages to Durable Shards for aggregation and buffering before being pushed to clients over WebSockets via a primary Durable Session.
  131. Data at Cloudflare just got a lot faster: Announcing Live-updating Analytics and Instant LogsIntroduced 'Live-updating Analytics' for Pro, Business, and Enterprise customers, providing real-time updates to analytics dashboards. Launched 'Instant Logs' for Enterprise customers, enabling immediate viewing of HTTP request logs in the dashboard with an average delay of two seconds, aiming for sub-second latency. This enhances visibility for security engineers to respond to threats and for developers to monitor configuration changes.
  132. Introducing logs from the dashboard for Cloudflare WorkersIntroduced dashboard-based logging and exception viewing for Cloudflare Workers, providing real-time access to `console.log` output and request headers. Enhanced `wrangler tail` command with advanced filtering (IP address, status, HTTP method, search query), sampling rates, and JSON output format for improved debugging and CI/CD integration. Added support for logging Durable Objects interactions.
  133. Automatic Remediation of Kubernetes NodesThis post introduces Sciuro, an open-source replacement for Node Problem Detector. Sciuro synchronizes Kubernetes node conditions with firing alerts in Alertmanager, enabling automated remediation of unhealthy nodes. It leverages existing Prometheus metrics and Alertmanager routing to update node conditions, which can then trigger remediation actions like node reboots. The post details a specific use case of automatically remediating nodes with ballooning Calico network interfaces by monitoring the number of interfaces via Prometheus and updating the node condition via Sciuro, which then triggers a reboot procedure.
  134. More products, more partners, and a new look for Cloudflare LogsIntroduced Firewall Events and Network Error Logging (NEL) Reports as new log datasets. Added support for S3-compatible APIs as a log destination. Enabled direct log push integrations with Microsoft Azure Sentinel, Splunk, Sumo Logic, and Datadog. Redesigned the Logpush configuration UI to select datasets and destinations.
  135. Improving your monitoring setup by integrating Cloudflare’s analytics data into Prometheus and GrafanaThis post details the creation of a custom Prometheus exporter that pulls data from Cloudflare's GraphQL Analytics API. The exporter transforms this data into Prometheus metrics, enabling integration with existing monitoring stacks like Prometheus and Grafana. It covers the design choices (Go, Cloudflare SDK), the specific GraphQL API nodes used (_http1mGroups, httpRequestsAdaptiveGroups_), the Prometheus metric formatting, and deployment strategies using Docker and Helm charts. It also provides a Grafana dashboard example for visualizing the data.
  136. Expanding the Cloudflare Workers Observability EcosystemThis post announces new partnerships with observability vendors (Sentry, New Relic, Datadog, Splunk, Sumo Logic, Honeycomb) to extend the observability ecosystem for Cloudflare Workers. It details how to integrate Workers applications with these platforms using tools like Toucan-js for Sentry and HTTP log endpoints for New Relic, Datadog, Splunk, and Sumo Logic. Specific Workers fields (WorkerCPUTime, WorkerStatus, WorkerSubrequest, WorkerSubrequestCount) are now available in HTTP logs for broader observability.
  137. Lessons Learned from Scaling Up Cloudflare’s Anomaly Detection PlatformThis post details the evolution of Cloudflare's Anomaly Detection platform for Bot Management, focusing on its scaling challenges and architectural improvements. It describes the initial monolithic service, its limitations, and the subsequent migration to a microservices architecture. Key technical details include the use of Histogram-Based Outlier Scoring (HBOS) for anomaly detection, ClickHouse for data storage, Redis with HyperLogLog for visitor behavior modeling, and Kafka for data ingestion and processing. Performance optimizations in Redis, such as binary key encoding and a recency register, are discussed. The move to microservices addressed issues with shared dependencies like ClickHouse and the unreliable Quicksilver integration, leading to a 10x reduction in load for baseline aggregation and better resource utilization.
  138. Soar: Simulation for Observability, reliAbility, and secuRityThis post introduces SOAR (Simulation for Observability, reliAbility, and secuRity), a new simulation system designed to test Cloudflare's complex, homogeneous edge environment. SOAR runs the same software stack as production data centers but in a controlled LAN environment with synthesized traffic. It addresses limitations of previous performance simulation systems by providing better isolation, coordination for complex scenarios (like Magic Transit failover), and engineer-friendliness through an interactive coordinator service callable from laptops and CI pipelines. The system was used to catch issues related to Magic Transit configuration and routing.
  139. Supercharging Firewall Events for Self-ServeIntroduced an overhauled Firewall Event log for Free, Pro, and Business customers, featuring a modal-free UI, expanded detail views with 'Additional matches' for multi-rule events, freeform search across all visible fields, and scoped date/time selection (24 hours for Free/Pro, 72 hours for Business).
  140. Improving request debugging in Cloudflare WorkersThis post introduces a new Network panel within the Cloudflare Workers inspector. This panel leverages the Chrome DevTools Protocol to expose the interaction between a Worker and its origin server, providing visibility into request and response headers, raw responses, timings, initiators, and the ability to export subrequests as HAR files. The implementation involved parsing DevTools Protocol messages, handling network-specific domains, and adapting the frontend to store response bodies on the client side.
  141. Logs from the EdgeIntroduced a Cloudflare Worker script that leverages sub-requests to capture HTTP request and response metadata (URL, method, headers, status, etc.) and asynchronously send it as JSON to an external ELK stack via HTTP PUT requests. This enables real-time traffic logging at the edge for all customers, regardless of their plan.
  142. Introducing ebpf_exporterThis post introduces ebpf_exporter, a tool that leverages eBPF to collect low-level system metrics, specifically focusing on providing histograms for event timings (e.g., disk I/O latency) which are superior to simple counters for understanding performance. It details the limitations of existing exporters like node_exporter and cAdvisor for detailed event timing analysis, explains the benefits of eBPF for low-overhead, safe, and universal kernel instrumentation, and demonstrates how BCC can be used to compile C code into eBPF bytecode for metric collection. The tool aims to provide deeper insights into system behavior than traditional counters.
  143. Tracing System CPU on Debian StretchThis post details the debugging process for high CPU usage and RCU stalls encountered during a Debian Stretch OS upgrade on Kafka nodes. It highlights the use of `perf top` for identifying CPU-bound processes and kernel logs (`dmesg`) for diagnosing memory allocation issues and RCU stalls. The investigation involved comparing upgraded nodes with control nodes, rolling back to a previous kernel and OS version, and using flamegraphs to pinpoint the issue to the Jessie to Stretch OS upgrade, specifically related to memory management and kernel message handling.
  144. Want to see your DNS analytics? We have a Grafana plugin for thatThis post introduces a Grafana plugin for Cloudflare's DNS analytics API. The plugin allows users to visualize DNS traffic, including response code breakdowns (e.g., NXDOMAIN during DDoS attacks), query types, and data center distribution. It also enables custom chart creation within Grafana dashboards and provides a pre-installed dashboard for Virtual DNS customers. The plugin is installed via `grafana-cli plugins install cloudflare-app` and uses the DNS analytics API for data retrieval.
  145. What CloudFlare LogsThis post clarifies Cloudflare's log retention policies, stating that most customer access logs are discarded within 4 hours, while Enterprise customers can opt to store logs for 3 days. Error logs are kept for 1 week. It also explains that aggregate data is generated for analytics and security threat detection, but raw logs are not retained for most customers, making it impossible to retrieve specific visitor data.