Data Infrastructure & Analytics

8/3/2026

Introduced the Billable Usage API, a new endpoint that provides programmatic access to Cloudflare account usage and cost data. The API returns data broken down by product and service period, aligning with the FinOps Open Cost and Usage Specification (FOCUS). This enables automated cost tracking, reporting, and integration with FinOps toolchains.

7/1/2026

This post introduces the Cloudflare Monetization Gateway, a new capability that allows customers to charge for any resource protected by Cloudflare using usage-based pricing and micropayments via the x402 protocol and stablecoins. This extends Cloudflare's data infrastructure and analytics by enabling granular metering and payment enforcement at the edge, addressing the evolving business model of the web driven by AI agents and the need for efficient, low-cost transaction rails for sub-cent payments.

5/28/2026

This post details the creation of Cloudflare's internal data platform, Town Lake, and an AI data agent, Skipper. Town Lake is a data lakehouse architecture using Apache Trino as the query engine, R2 for object storage with Apache Iceberg for metadata, DataHub for metadata cataloging, Lifeguard for access control, Skimmer for PII detection using Workers AI, Transformer for ELT on Workflows, and a custom ingestion orchestrator. Skipper leverages Town Lake, Workers AI, Durable Objects, D1, R2, and Workflows to translate natural language questions into SQL queries, package results into dashboards, and assist with data transformation and access control.

5/27/2026

This post details the use of Cloudflare Radar data to analyze and confirm the partial restoration of Internet access in Iran following a prolonged shutdown. It examines traffic volume, regional distribution, network provider impact, and DNS query increases, providing specific metrics and observations. It also highlights the continued impact on IPv6 connectivity compared to IPv4, suggesting the shutdown was achieved through application filtering or whitelisting rather than routing table manipulation.

5/14/2026

This post details a critical performance bottleneck encountered in Cloudflare's petabyte-scale ClickHouse analytics platform, specifically impacting the billing pipeline. The issue stemmed from a new partitioning scheme ('namespace', 'day') which, while enabling per-namespace retention, led to a massive increase in data parts and subsequent query planning overhead and lock contention. The post outlines the investigation using flame graphs and the development of three patches: using shared locks, deferring vector copying, and optimizing part filtering, which significantly improved performance and were contributed upstream to ClickHouse.

5/12/2026

This post details a critical bug discovered in the CUBIC congestion control algorithm, specifically how a Linux kernel optimization, when ported to Cloudflare's quiche QUIC implementation, caused the congestion window to get permanently pinned at its minimum after a congestion collapse event. The post explains the root cause, tracing it back to an optimization for handling application idle periods and how its implementation in user-space QUIC differed from the kernel's TCP, leading to unexpected oscillations and failure to recover bandwidth. It highlights the importance of rigorous testing in edge cases and the successful resolution of the bug.

4/16/2026

This post introduces Artifacts, a new primitive: a distributed, versioned filesystem built for agents that speaks Git. It allows programmatic creation of Git repositories for agents, sandboxes, and other compute paradigms, with REST API and native Workers API access. It also introduces ArtifactFS, a filesystem driver for mounting large Git repos quickly by hydrating file contents on the fly, ideal for agents and sandboxes with critical startup times. Artifacts is built on Durable Objects and a custom Git server implementation in Zig compiled to Wasm.

3/3/2026

This post introduces a significant evolution of Cloudflare's Threat Intelligence Platform (TIP), moving away from complex ETL pipelines to a sharded, SQLite-backed architecture running GraphQL directly on the edge. This enables real-time visualization and automation of threat response with sub-second query latency over millions of events. The platform unifies global telemetry with analyst investigations to provide a single source of truth for proactive threat blocking. It complements SIEMs by providing specialized schema and long-term retention for deep adversary tracking, enriching raw logs with historical actor patterns and enabling faster, more accurate decision-making. The architecture eliminates bottlenecks through parallel execution at the edge using SQLite-backed Durable Objects and Cloudflare Workers, with dynamic visualizations like Sankey Diagrams and attribute mapping for correlating threat events. Saved configurations and real-time notifications facilitate proactive threat hunting, and automated rules and STIX2 exports enable seamless integration with SIEM/SOAR platforms.

2/27/2026

Introduced a new ASPA deployment monitoring feature within Cloudflare Radar to track the adoption and trends of the ASPA standard across Regional Internet Registries (RIRs) and Autonomous Systems (AS). Detailed the technical implementation and validation process of ASPA, including its role in securing Internet routing by verifying the AS_PATH and preventing route leaks. Provided a practical example of creating ASPA objects within RIPE and ARIN dashboards.

2/3/2026

This post introduces R2 Local Uploads, a new feature for Cloudflare R2 object storage that significantly improves global upload performance. It allows object data to be written to a storage location close to the client first, then asynchronously copied to the bucket's primary location. This reduces upload request duration by up to 75% for cross-region uploads and leverages Cloudflare Queues for managing the asynchronous replication tasks.

12/18/2025

This post introduces significant enhancements to R2 SQL by adding support for aggregation queries (GROUP BY, SUM, COUNT, AVG, etc.). It details the implementation of scatter-gather and shuffling aggregation techniques to efficiently process these queries over large datasets stored in R2 Data Catalog. This enables users to perform complex analytical tasks like generating reports, identifying trends, and finding anomalies directly within R2, building upon existing filter query capabilities.

12/15/2025

This post introduces Cloudflare's 'Top Internet Services of 2025' report, which analyzes DNS trends to rank the popularity of various internet services across nine categories. It details the methodology, which uses anonymized DNS query data from 1.1.1.1 and a machine-learning-assisted ranking method. The report highlights key trends such as the rise of Asian e-commerce, the emergence of ChatGPT competitors like Claude and Gemini, shifts in social media platforms like Instagram and TikTok, and the growing influence of AI in news consumption. It also provides a detailed breakdown of the top services within each category and discusses regional variations.

12/15/2025

This post introduces the 2025 Cloudflare Radar Year in Review, detailing observed Internet trends. Key contributions include data on global traffic growth, AI crawling patterns, post-quantum encryption adoption, connectivity metrics, and security threats. It also highlights new data sets for speed tests and DDoS attack sizes, and specific AI model usage on Workers AI.

11/3/2025

This post details the use of Cloudflare Radar's historical data on TCP connection resets and timeouts to corroborate reports of IP unblocking and potential firewall testing in Turkmenistan. It analyzes trends in request volume and TCP anomaly patterns across different autonomous systems within Turkmenistan to provide insights into network behavior changes.

10/29/2025

This post shares aggregate insights about TCP connection characteristics observed through Cloudflare's global CDN. It details the methodology for collecting and analyzing socket-level metadata from the Linux kernel's TCP_INFO struct, focusing on connections that close gracefully with a FIN packet and have at least one successful HTTP request. The analysis presents empirical data on packet counts (response packets, and packet ratios between client and server) and bytes sent, highlighting the heavy-tailed nature of internet traffic and differences across HTTP protocol versions. This data is intended to inform simulations and provide a deeper understanding of real-world network behavior.

10/27/2025

This post introduces the scientific practice of Internet measurement as a critical component of Cloudflare's data infrastructure and analytics. It highlights the inherent opacity of the Internet and the unique challenges it presents for measurement. The post details key measurement concepts like active vs. passive and direct vs. indirect measurement, and outlines the measurement lifecycle (data curation, modeling, validation). It uses an internal example of analyzing HTTP traffic spikes in Ukraine to illustrate the importance of context and avoiding erroneous conclusions, emphasizing the need for rigor, repeatability, and reproduction in measurement.

10/27/2025

Introduces a new TLD page on Cloudflare Radar, providing insights into TLD popularity, activity, and security. This includes a ranking of top-level domains based on DNS Magnitude, a metric that estimates a domain's overall visibility on the Internet by incorporating the number of unique clients querying domains within a TLD. The post also details the methodology behind DNS Magnitude calculation and the types of TLDs (gTLD, grTLD, ccTLD, iTLD, sTLD). Additionally, it describes dedicated per-TLD pages offering general information (type, manager, DNS magnitude, DNSSEC/RDAP support) and WHOIS data, with direct links to Cloudflare Registrar for supported TLDs.

10/27/2025

This post details foundational concepts and tradeoffs in Internet measurement, discusses the evolution of Cloudflare Radar and the Internet speed test, and introduces a framework for Internet resilience. It also covers advancements in post-quantum cryptography, including Merkle Tree Certificates, and explores optimizations in Cloudflare's Linux networking stack for addressing space and performance. Additionally, it details methods for CGNAT detection, DDoS defense, and cache efficiency. The post also introduces cryptographic protocols for an agentic web, building on Web Bot Auth, to enable secure identification of automated clients and protect websites from various threats, with a focus on post-quantum readiness.

10/27/2025

This post details the evolution of Cloudflare Radar over five years, highlighting its expansion into Internet security transparency with Certificate Transparency (CT) data, TCP resets and timeouts for detecting connection tampering, and post-quantum encryption adoption tracking. It also introduces AI Insights, providing visibility into AI crawler activity, crawl-to-refer ratios, and robots.txt usage by AI bots. Furthermore, Radar has enhanced routing visibility with data on route leaks and origin hijacks, contributing to Internet resilience.

9/26/2025

Introduced AS-SET information to Cloudflare Radar's routing section, providing a public listing for network operators. This feature allows monitoring of AS-SET memberships, helps in building BGP route filters using tools like bgpq4, and demonstrates how accurate AS-SETs prevent route leaks. The feature also highlights AS-SET misuse and provides enhanced AS-SET data with inferred ASN, IRR sources, member counts, AS Cone, and upstream information. A tree-view visualization for AS-SET inclusion is also available.

9/26/2025

Introduced regional traffic insights to Cloudflare Radar, enabling localized analysis of Internet traffic trends. This includes breaking down traffic by bytes, requests, desktop/mobile device shares, and bot/human traffic shares. Also added detailed Certificate Transparency (CT) data to Radar. Enhanced the Data Explorer to allow for combined regional and ASN analysis, and added new summary and time series views for regional traffic characteristics on country-level pages.

9/25/2025

This post announces the launch of the Cloudflare Data Platform, a comprehensive solution for analytical data. It introduces Cloudflare Pipelines for event ingestion and transformation using SQL, R2 Data Catalog with enhanced features like automatic compaction for managed Apache Iceberg metadata, and R2 SQL for petabyte-scale querying. This significantly expands Cloudflare's offerings in data infrastructure by providing a complete, integrated platform for analytical data, building upon previous work with R2 Object Storage and R2 Data Catalog.

9/25/2025

This post introduces R2 SQL, a new serverless query engine that allows users to run SQL queries directly over petabytes of data stored in R2 object storage. It details the architecture, including the Query Planner that uses R2 Data Catalog (Apache Iceberg) metadata (partition-level and column-level stats) to prune irrelevant data, and the Query Execution system that leverages Cloudflare's global network and Workers for parallel processing. Key innovations include a streaming planning pipeline for early execution and a 'stop early' strategy for queries with ORDER BY clauses, enabling faster results by processing prioritized data and proving completion without scanning the entire dataset.

8/5/2025

This post details the optimization of a critical low-latency operation within Cloudflare's privacy proxy product. By analyzing tracing and metrics data, engineers identified a 40ms latency bottleneck in double-spend checks for Privacy Pass tokens. Through detailed investigation and hypothesis testing, they pinpointed the root cause to be the interaction of Nagle's algorithm and delayed ACKs in TCP, exacerbated by the Linux kernel's HZ setting. The fix involved implementing a `BufWriter` to buffer writes and send commands as single messages, reducing latency to under 1ms. This demonstrates a deep dive into network protocol optimization for performance-critical services.

7/23/2025

This post introduces Jetflow, a new framework developed by Cloudflare's Business Intelligence team for building performant and efficient data pipelines. Jetflow addresses challenges with petabyte-scale data lakes and high-volume daily ingestion, achieving over 100x efficiency improvements in GB-s and >10x performance improvements in row ingestion rates. Key technical contributions include a modular design with Consumers, Transformers, and Loaders, support for configuration-as-code, and the use of Arrow as an in-memory columnar data format for efficient data transfer and reduced GC overhead. The framework also optimizes data processing by reading data in columnar formats and writing Parquet files directly from Arrow columns.

7/16/2025

Introduces the integration of marimo, an open-source reactive Python notebook, with Cloudflare's data platforms. This includes built-in Cloudflare authentication for notebooks, open-source notebook examples for exploring Cloudflare services (R2, Workers AI, D1), and the ability to deploy marimo notebooks on Cloudflare Containers for scalable data workflows. The post details how marimo's reactive execution model and Python-native storage improve reproducibility and deployment compared to traditional notebooks.

7/8/2025

This post details the adoption of TimescaleDB, a PostgreSQL extension, for the analytics and reporting needs of the Digital Experience Monitoring (DEX) product within the Zero Trust suite. It highlights the decision-making process, favoring simplicity and a 'daily driver' approach over the complexity of ClickHouse for initial MVP launch. The post explains the design of the configuration and analytics planes, the use of PostgreSQL for storing structured logs, and the optimization of multicolumn indexes for time-series data, showcasing a pragmatic approach to scaling analytics by leveraging familiar technologies and strategic use of extensions.

6/27/2025

This post details the impact of ISP-level throttling in Russia on Cloudflare's network and users. It analyzes traffic trends using Cloudflare Radar, showing a significant decrease in traffic served to Russian users. It also presents Network Error Logging (NEL) data indicating increased TCP resets and QUIC protocol errors. Furthermore, internal data analysis using packet loss metrics and TCP resets and timeouts (specifically the 'Post PSH' stage) corroborates the throttling mechanism, which limits data transfer to 16 KB.

6/13/2025

This post details a significant service outage on June 12, 2025, caused by a failure in the underlying storage infrastructure for Workers KV, a critical dependency for many Cloudflare products. The outage impacted a wide range of services including Workers KV, WARP, Access, Gateway, Images, Stream, Workers AI, Turnstile, AutoRAG, Zaraz, and parts of the Dashboard. The incident highlighted the fragility of relying on a central data store for Workers KV and the cascading effects of such failures across the platform, emphasizing the urgent need for the ongoing transition to more resilient infrastructure.

5/21/2025

Introduced a new real-time BGP route lookup service accessible through the Cloudflare Radar web interface and API. This service provides visualizations of real-time routes on IP prefix pages, including Sankey diagrams illustrating routes to Tier 1 networks and options to show full paths. It also displays prefix origin tables with visibility percentages and RPKI validation outcomes. The Cloudflare Radar API now offers programmatic access to this real-time BGP route data, enabling automated tools and systems.

5/20/2025

This post introduces new tools and techniques for analyzing performance data, revealing surprising insights into system behavior and the impact of statistical aggregation on understanding true latency. It also clarifies definitions for performance metrics like TTFB and discusses the limitations of aggregated data in representing actual latency.

4/28/2025

This post details the impact of a major power outage in Portugal and Spain on internet traffic, network quality, and routing, providing granular data analysis from Cloudflare's global network perspective. It includes country-level, network-level, and regional traffic drops, as well as changes in network quality (download speeds, latency) and routing (announced IP address space).

4/10/2025

This post announces the open beta of Cloudflare Pipelines, a new streaming ingestion product that allows users to ingest high volumes of structured, real-time data directly into R2 object storage. It also announces the acquisition of Arroyo, a cloud-native, distributed stream processing engine, to enable real-time data transformation and loading into open table formats. The post details the architecture of Pipelines, which uses Durable Objects and embedded SQLite for scalable ingestion and batching into R2, and explains how it integrates with Workers and R2 to form a data platform. It also provides examples of how to create and use a pipeline.

4/10/2025

This post introduces the R2 Data Catalog, a managed Apache Iceberg catalog built directly into Cloudflare R2 buckets. This feature simplifies querying large analytic datasets stored in object storage by providing database-like capabilities (ACID transactions, schema evolution) without the overhead of managing external data catalogs or incurring egress fees. It also details Apache Iceberg's functionality and how it works with object storage, and provides a guide for creating the first Iceberg table on R2.

4/2/2025

This post details a deep dive into a persistent testing failure within Cloudflare's internal Backstage instance on Linux, characterized by a consistent 27-second timeout leading to a cryptic '[Error]'. The troubleshooting process involved extensive use of shell history, Jest flags, and ultimately `strace`, revealing an underlying system-level issue related to `epoll_wait` and `SIGCHLD` signals, potentially linked to older system components or configurations. This investigation highlights the challenges of maintaining robust development environments and the importance of low-level system analysis for debugging.

3/25/2025

This post details a significant incident impacting R2 object storage and several dependent services (Cache Reserve, Images, Log Delivery, Stream, Vectorize) due to a credential rotation error. It highlights the critical need for robust visibility into authentication mechanisms and proper environment management in production systems, directly impacting the reliability and availability of data infrastructure and analytics services.

3/18/2025

This post introduces several new datasets and visualizations to Cloudflare Radar's security insights: new DDoS-focused graphs with geographic and AS-level distribution maps, new insights into leaked credential trends with a dedicated graph showing clean vs. compromised requests, and a new dedicated Bots page for monitoring bot traffic vs. human traffic. It also details the refactoring of the Security & Attacks page into separate Application Layer and Network Layer sections.

2/27/2025

Introduced a dedicated DNS page on Cloudflare Radar, leveraging data from the 1.1.1.1 resolver and the AS112 Project. This page provides detailed insights into DNS traffic trends, including global, location, and ASN traffic, protocol usage (UDP, DoT, DoH, TCP), query and response characteristics (record types like A, AAAA, HTTPS), and DNSSEC usage. The post details the analysis of this data, including its use for identifying ccTLD popularity and corroborating reported Internet outages or blocking of 1.1.1.1.

2/14/2025

This post delves into the Linux kernel's 'hung task' warnings, explaining their meaning, how the kernel identifies them, and how Cloudflare configures monitoring for these events. It provides detailed examples of debugging hung tasks related to XFS filesystem issues and coredump generation, highlighting how these warnings can surface underlying system performance problems or unexpected behaviors, thereby contributing to the broader understanding and optimization of Cloudflare's data infrastructure.

2/13/2025

Introduced Automatic Audit Logs, a new system that standardizes and automates audit log generation across Cloudflare products. This system leverages an API Gateway, Audit Log Redactor Workers, OpenAPI schemas for data redaction, and an Ingestor service to process and enrich logs. Key technical details include streaming HTTP requests via RPC, using OpenAPI schemas to identify and redact sensitive information, and a data pipeline involving Logfwdr, Logreceiver, and Buftee buffers. The Ingestor service transforms requests into standardized audit log records, and logs are enriched with user and authentication details before being stored. The system currently stores 30 days of logs, with plans to extend this to 18 months.

2/7/2025

This post details a significant incident where human error and insufficient validation safeguards during abuse remediation led to the disabling of the R2 Gateway service, impacting R2 object storage and several dependent services including Stream, Images, Cache Reserve, Vectorize, and Log Delivery. It highlights the critical role of R2 as a data infrastructure component and the cascading effects of its unavailability, while also emphasizing that no data was lost or corrupted. The incident underscores the importance of robust controls and validation in managing production data services.

2/4/2025

Introduced a new dedicated "AI Insights" page on Cloudflare Radar, expanding on existing AI bot and crawler traffic graphs. The new page incorporates additional metrics to understand AI-related trends from multiple perspectives, including: relative popularity of publicly available Generative AI services based on 1.1.1.1 DNS resolver traffic, usage of robots.txt directives to restrict AI bot access to content, and open source model usage as seen by Cloudflare Workers AI. The AI Insights page provides visibility into AI bots and crawlers traffic trends, the popularity of Generative AI services, analysis of robots.txt files for AI user agents, and the popularity of models and tasks on Workers AI.

1/27/2025

This post details Cloudflare's techniques for handling over 700 million events per second for its analytics products. It introduces 'bottomless buffers' for controlled data downsampling using max-heaps and max-min fairness to manage buffer overflow. It also explains adaptive sampling in Logreceiver to improve accuracy for small customers and aggressively downsample large ones. The post elaborates on using the Horvitz-Thompson estimator with Poisson sampling to derive analytics (SUM, COUNT, AVG) from downsampled data and calculate confidence intervals, addressing challenges like unknown population size.

1/21/2025

This post analyzes the impact of the US ban on TikTok on DNS traffic, detailing the timeline of traffic drops and recovery. It also tracks the surge in DNS traffic for TikTok alternatives like RedNote (Xiaohongshu) in the US and globally, providing country-specific growth percentages and correlating these trends with app store rankings. The analysis leverages data from Cloudflare's 1.1.1.1 DNS resolver and autonomous system-level data.

1/19/2025

This post details the real-time impact of the US TikTok ban on DNS and autonomous system traffic, showing significant drops for TikTok and surges for alternatives like RedNote. It also analyzes traffic shifts in other countries and the subsequent service restoration and partial recovery of TikTok traffic.

12/9/2024

This post details the 2024 Cloudflare Radar Year in Review, highlighting key findings across traffic, adoption & usage, connectivity, security, and email security. It introduces new metrics for AI bot & crawler traffic, search engine and browser market share, connection tampering, and "most dangerous" top level domains (TLDs). The review also includes year-over-year and geographic comparisons for selected metrics, and provides insights into global Internet traffic growth, popular services, Starlink traffic growth, AI crawler activity, TLS 1.3 adoption with post-quantum encryption, mobile device traffic share, HTTP/2 and HTTP/3 usage, popular website technologies and programming languages, search engine and browser market share, Internet disruptions, IPv6 adoption, Internet speed, mobile traffic share, TCP connection terminations, mitigated malicious traffic, bot traffic origins, attacked industries, Log4j vulnerability persistence, routing security improvements, malicious email rates, and the prevalence of spam/malicious emails from specific TLDs.

12/9/2024

This post details the 2024 Cloudflare Radar Year in Review, presenting rankings of top Internet services globally and by region. It leverages anonymized DNS query data from 1.1.1.1 to analyze trends in categories like Generative AI, Social Media, E-commerce, and more. Key findings include ChatGPT's continued growth and its entry into the top 50 overall, the rise of services like Temu and GitHub Copilot, and regional variations in service popularity. The analysis highlights shifts in user preferences and the impact of major events on internet traffic patterns.

12/3/2024

This post analyzes Cyber Week 2024 internet trends, detailing global and regional traffic patterns, mobile vs. desktop usage shifts, e-commerce DNS trends, and the growth of cyber threats like DDoS attacks. It leverages aggregated HTTP requests, DNS queries, and DDoS mitigation data to provide these insights, comparing them to previous years and highlighting country-specific variations.

10/1/2024

This post details the analysis of a Verizon mobile network outage on September 30, 2024, using Cloudflare Radar data. It quantifies the impact on HTTP request volume from AS6167 (Verizon's mobile network ASN), showing a decline of up to 9% below expected levels. The analysis also breaks down traffic impact by specific impacted cities, highlighting significant drops in Omaha (approx. 30%) and other cities (10-20%), while noting some cities like New York and Atlanta showed week-over-week increases. The post concludes that voice services may have been more impacted than data services, based on the observed traffic patterns.

9/27/2024

Introduced the Cloudflare Radar Data Explorer, a web-based interface for building complex API queries and visualizing results. Launched an AI Assistant that uses Cloudflare Workers AI to translate natural language questions into Radar API calls, with results displayed in the Data Explorer. Detailed the technical implementation of the AI Assistant using prompt engineering and multiple inference calls to LLMs, and the Data Explorer's visualization logic based on API response content.

9/20/2024

This post introduces 'Scenario Planner', a new system that enables sophisticated "what-if" capacity planning for Cloudflare's global network. It models future infrastructure needs by simulating various demand (customer growth, traffic shifts) and supply (data center failures, server additions) scenarios, translating them into a common unit of 'CPU Time'. The system outputs heatmaps and expected failover views to guide proactive infrastructure decisions, ensuring continued service availability and performance.

9/11/2024

This post details the analysis of Internet traffic patterns and security trends during the 2024 US Presidential debates, showing significant traffic drops in specific states and increased DNS traffic to candidate and news-related domains. It also highlights the impact of specific events like Taylor Swift's endorsement on traffic.

9/10/2024

This post introduces the 'trie-hard' open-source Rust crate, developed to optimize the `clear_internal_headers` function within Cloudflare's pingora-origin service. This optimization significantly reduces CPU utilization by over 1.28% by employing a custom trie data structure for efficient header removal, demonstrating a focus on micro-optimizations within core network infrastructure to improve overall performance and handle increased traffic.

9/5/2024

This post introduces new capabilities for detecting and analyzing global connection tampering. It details the methodology for identifying connection tampering signatures by analyzing abrupt connection closures and protocol anomalies. The post presents findings on the prevalence of connection tampering across different countries, correlating these findings with existing research and identifying potential causes like censorship and zero-rating. It also announces a new dashboard and API on Cloudflare Radar for near real-time visibility into connection tampering events.

9/5/2024

Introduced a new dashboard and API endpoint on Cloudflare Radar to track and analyze anomalous TCP connections (resets or timeouts within the first 10 ingress packets). This involved developing detection mechanisms to classify these connections and provide insights into network issues, scanning, and attacks. The post details the classification of connection stages (Post-SYN, Post-ACK, Post-PSH, Later) and provides guidance on interpreting the data.

8/23/2024

This post analyzes internet traffic, DNS trends, and cyberattacks related to the 2024 Democratic National Convention. It details traffic patterns in Chicago, growth in DNS traffic to Kamala Harris-related sites and fundraising domains, increased news consumption during the convention, and observed DDoS attacks targeting politically-related organizations. It also examines email trends involving political candidates, noting surges in Trump-related spam and malicious messages.

8/13/2024

This post introduces HTTP request traffic as a new metric on Cloudflare Radar, complementing existing byte-based traffic graphs. It details the implementation of this new metric, including its integration into the Overview and Traffic sections of Radar, and explains the value of request-based insights for distinguishing real events from data pipeline issues and for better reflecting user activity, especially with the rise of API traffic. The post also clarifies the difference between bytes and requests, using Portugal's traffic patterns as an example.

8/1/2024

This post details the analysis of widespread Internet disruptions in Bangladesh, Syria, and Pakistan using Cloudflare Radar. It showcases the tracking of traffic anomalies, IP address space changes, and internet quality metrics during these events, providing specific examples of how Radar data was used to understand the scope and impact of government-imposed shutdowns and accidental infrastructure failures.

7/30/2024

This post details the analysis of Internet traffic patterns during the Paris 2024 Summer Olympics, correlating traffic drops with specific ceremony moments and popular sporting events. It also analyzes DNS trends to official Olympic websites by country and highlights the shift in interest from France to the US. The analysis leverages Cloudflare's global network data and Cloudflare Radar's reporting capabilities.

7/22/2024

This post details the analysis of pre-event trends for the Paris 2024 Olympics, including DNS traffic patterns by country and spikes related to ticket sales and news announcements, as well as an analysis of email security trends (spam and malicious emails) related to the event. It leverages data from 1.1.1.1 resolver and Cloudflare's Email Security service to provide insights into user interest and potential security threats.

7/19/2024

This post details the analysis of internet traffic and cyberattacks surrounding the 2024 U.S. Republican National Convention and the attempted assassination of Donald Trump. It presents data on DDoS attacks targeting political websites, traffic surges to news outlets following the assassination attempt, and specific traffic increases to Republican party and fundraising websites during the RNC. It also highlights a DDoS attack against a think tank during the RNC. The post emphasizes the short duration of most DDoS attacks and the need for automated mitigation systems. It also points to Cloudflare Radar and its 2024 Elections Insights report for further trend analysis.

7/11/2024

This post analyzes the impact of UEFA Euro 2024 on Internet traffic, focusing on finalist countries Spain and England. It details how major football matches, especially those broadcast on national TV and featuring critical moments, lead to significant drops in Internet traffic. The analysis highlights distinct patterns for Spain (drops during games against major teams) and England (drops during crucial late-game moments and knockout stages). This contributes to understanding large-scale event impacts on network traffic patterns.

7/8/2024

This post analyzes internet traffic shifts and cyberattack trends during the 2024 French legislative elections. It details DDoS attacks targeting political parties, including peak request rates and total requests. It also examines internet traffic drops correlated with election result announcements, regional traffic variations, and changes in mobile device usage share. Additionally, it analyzes DNS trends for news outlets, TV domains, and social media platforms during the election periods.

6/28/2024

This post details how Cloudflare Radar analyzed the impact of the first 2024 US presidential debate on internet traffic and security trends. It quantifies traffic drops at the state level, analyzes DNS query surges for political websites and donation platforms, and examines email security data related to political campaigns. The analysis leverages Cloudflare's global network data and email security service to provide insights into user behavior and threat landscapes during significant political events.

6/21/2024

This post details how Cloudflare's global network data is used to analyze the impact of major events like the UEFA Euro 2024 football tournament on local Internet traffic, observing drops in traffic during games and spikes in social media usage during breaks. It also highlights observed DDoS attacks during the event, including specific metrics like requests per hour and requests per second for a significant attack on a translation tool.

6/10/2024

This post details the analysis of internet traffic trends and cybersecurity insights observed during the 2024 European Parliament election. It includes data on traffic drops during voting hours, increases as results were announced, and specific domain traffic growth for election-related sites. It also highlights government-focused cyberattacks observed during the election period, specifically mentioning DDoS attacks on Dutch political websites.

6/7/2024

This post details the analysis of internet traffic patterns and cyberattack trends during national elections in the Netherlands, South Africa, Iceland, India, and Mexico. It observes traffic dips during polling hours and spikes when results are announced, and notes DDoS attacks targeting political websites. Specific examples include DDoS attacks on Dutch political websites during the European Parliament election, traffic drops and spikes during Mexico's general election, and 44 days of traffic dips and mobile spikes during India's general election. The post also highlights the use of Cloudflare Radar for tracking these events and provides links to related reports and previous election analyses.

4/14/2024

This post details the analysis of internet traffic patterns in Israel and Palestinian territories during the April 13, 2024, Iranian attack. It observes traffic surges in Israel correlated with the attack and a drop in traffic in Palestinian territories. It also analyzes traffic patterns in Iran, linking early morning declines to the conclusion of Ramadan and noting a subsequent increase. The post also reports on application layer attacks, noting a lack of significant changes targeting Israel and a historical trend of increased attacks on the Government Administration sector.

4/9/2024

This post analyzes the impact of the Great North American Eclipse on internet traffic across Mexico, the US, and Canada. It presents data showing significant drops in bytes delivered and HTTP requests, correlating these drops with the path of totality. The analysis includes state-level breakdowns for the US, country-level data for Mexico and Canada, and visual representations of traffic changes over time. It highlights the platform's capability to track and analyze large-scale, event-driven traffic anomalies.

4/3/2024

Introduced Event Notifications for R2, allowing data change events to trigger Cloudflare Workers via Queues. Enhanced Super Slurper to support migrations from Google Cloud Storage (GCS) to R2. Launched a private beta for an Infrequent Access storage tier in R2, with associated pricing for storage, operations, and data retrieval, while maintaining zero egress fees.

3/8/2024

Introduced Log Explorer, a new feature that allows users to investigate HTTP and Security Event logs directly within the Cloudflare Dashboard. Log Explorer is built on Cloudflare R2 and leverages the Delta Lake protocol for ACID-compliant storage and querying. It provides a SQL interface for analyzing raw log data, enabling detailed security investigations and reducing the need for third-party log storage solutions. This enhances the capabilities of Security Analytics by providing granular log access.

2/12/2024

This post details how Cloudflare Radar uses DNS name resolution data from the 1.1.1.1 resolver to estimate traffic to websites and analyze trends during specific events like the Super Bowl. It examines the traffic spikes driven by advertisements, the impact on categories like food delivery, social media, and sports betting, and local traffic trends in Kansas City and San Francisco. It also looks at email threat volume in the weeks leading up to the game.

1/24/2024

This post introduces 'Foundations,' an open-source Rust library designed to simplify the development of distributed, production-grade systems. It addresses key challenges in observability (logging, tracing, metrics), configuration, and security, drawing from Cloudflare's experience with its Oxy proxy framework. The library offers modularity, API ergonomics, and simplified setup, with specific enhancements in tracing (sampling flexibility, distributed trace stitching, trace forking) and logging (implicit context propagation). It also integrates with the Prometheus client for metrics and provides a procedural macro for easier metric definition.

12/12/2023

This post introduces a new category for Generative AI in the Cloudflare Radar rankings and details the methodology for ranking Internet services based on anonymized DNS query data from 1.1.1.1. It analyzes trends in Generative AI, Social Media, E-commerce, and other categories, highlighting the growth of services like OpenAI and Temu, and the shifts in social media platforms like X/Twitter and Threads. The post also discusses the relative popularity of top Internet services like Google, Facebook, and Apple, and the impact of major global events on news consumption.

12/12/2023

This post details the findings of the Cloudflare Radar 2023 Year in Review, analyzing global Internet traffic growth (25%), popular services (Google, OpenAI, Binance), mobile device traffic share (Android dominance), Starlink traffic growth, popular web technologies (Google Analytics, React, HubSpot), HTTP/2 and HTTP/3 usage, and the most popular language for API requests (NodeJS). It also covers connectivity and speed metrics, including Internet outages, IPv6 adoption rates, download speeds, and mobile device traffic share. Security insights include mitigated malicious traffic percentages, bot traffic origins, attacked industries, notable vulnerabilities (Log4j, HTTP/2 Rapid Reset), post-quantum encryption adoption, email threats, and RPKI validation improvements. The post highlights the evolution of Cloudflare Radar since its launch in 2020 and its role in providing unique near-real-time perspectives on Internet patterns.

11/28/2023

This post details Cloudflare's analysis of Internet traffic and e-commerce trends during Cyber Week (Black Friday and Cyber Monday). It provides specific metrics on peak HTTP requests per second, daily request volumes, and DNS query volumes for both the 1.1.1.1 resolver and authoritative DNS servers. It also breaks down traffic by country and device type (mobile vs. desktop), and analyzes DNS trends for e-commerce sites, including category-specific insights (electronics, fast fashion, second-hand). Finally, it touches upon cyber threat trends, specifically DDoS attacks during this period.

11/24/2023

This post analyzes Thanksgiving holiday internet traffic trends in the US using Cloudflare Radar data. It details hourly and daily traffic drops, state-by-state variations in traffic reduction, the increase in mobile traffic percentage, DNS traffic patterns for food delivery and online grocery services, DDoS attack trends, and email message volume and spam rates. The analysis highlights the consistency of these trends year-over-year and the impact of holidays on internet behavior.

10/9/2023

This post details the analysis of internet traffic patterns and cyberattacks in Israel and Palestine following the October 2023 attacks. It quantifies traffic surges in Israel and network outages and traffic drops in Palestine, including specific Autonomous System (AS) numbers affected. It also analyzes the scale and timing of DDoS attacks targeting both regions, providing metrics like requests per second and daily blocked requests. The post highlights the use of Cloudflare Radar data to monitor these events and provides links to real-time data for continued observation.

9/26/2023

This post introduces the integration of an internal traffic anomaly detection and alerting tool into Cloudflare Radar's Outage Center (CROC). It details how country and network-level traffic anomalies are now displayed on CROC and made available via API. Additionally, new Radar notification functionality is launched, allowing users to subscribe to alerts for traffic anomalies, confirmed Internet outages, route leaks, or route hijacks at country or autonomous system levels, leveraging the Cloudflare dashboard's existing notification system. The post also provides technical details on the traffic anomaly detection methodologies and includes example API requests and responses for traffic anomalies.

9/26/2023

This post details the technical approach to detecting Internet outages and traffic anomalies for Cloudflare Radar. It introduces a new system for automatically identifying anomalous traffic events (primarily traffic drops) across countries and Autonomous Systems (ASes). The system uses time-series analysis of various data sources (DNS, HTTP, NetFlows, NEL) to detect deviations from expected patterns. It outlines the process of verifying these anomalies, categorizing them as 'Verified' or 'False Positive', and the challenges of modeling diverse traffic patterns, including seasonality, trends, and external events like holidays. The post also discusses the criteria for selecting entities for anomaly detection based on traffic signal strength.

8/31/2023

This post details the new features and insights introduced to Cloudflare Radar in 2023, including the URL Scanner, Internet Quality page, Trending Domains, and Routing page. It also highlights general Internet insights observed throughout the year, such as the impact of the war in Ukraine, the Virgin Media outage, holiday trends, and the effects of conflicts in Sudan and Niger on Internet patterns, as well as the coronation of King Charles III.

7/27/2023

Introduced a new 'Routing' page on Cloudflare Radar, providing detailed monitoring of BGP messages and related internet infrastructure data. This includes statistics on ASes, prefixes, and RPKI validation, as well as anomaly detection for BGP route leaks, hijacks, and RPKI invalid MOAS events. The page also offers insights into network connectivity (upstream, downstream, peering) and IP prefix origination. The data is made available through a developer API.

7/24/2023

Introduced 'Trending Domains' to Cloudflare Radar, providing 'Trending Today' and 'Trending This Week' lists. This feature leverages aggregated 1.1.1.1 resolver data and the existing Radar Domain Rankings to identify domains with increasing popularity. The methodology involves analyzing rank volatility across different list sizes and comparing daily ranks against historical data (best rank of previous four days for 'Trending Today', weighted average for 'Trending This Week') to quantify popularity surges. This enhances the granularity of internet traffic insights.

6/23/2023

Introduced the Cloudflare Radar Internet Quality page, which provides country and network-level insights into Internet connection performance (bandwidth) and quality (latency, jitter) over time. This page leverages aggregated results from the speed.cloudflare.com speed test tool and measurements against a set of Cloudflare & third-party targets (Internet Quality Index - IQI). The post details the methodology for calculating IQI and Connection Quality, including the use of trimeans and population weighting, and explains how to navigate the new page.

6/13/2023

This post details the use of Cloudflare Radar to analyze internet shutdowns in Iraq and Algeria for exam purposes. It showcases how Radar data (traffic, BGP announcements, 1.1.1.1 resolver usage) can be used to identify and characterize these disruptions, including differentiating between routing-based shutdowns and content blocking, and observing regional variations in impact. The post also highlights the integration of interactive graphs directly into the blog post using a new Cloudflare Radar feature.

5/19/2023

This post announces a major update to Cloudflare D1, its native serverless SQL database. Key contributions include: significant performance and scalability improvements with a new storage backend (up to 20x faster reads, 6.8-11x faster writes), enhanced developer experience with a new console interface, JSON function support, and location hints. It also outlines the pricing model, including an always-free tier and usage-based billing, and introduces 'Time Travel' for point-in-time recovery (last 30 days). This post marks a significant maturation of D1 from alpha to a more robust and performant offering.

5/16/2023

This post introduces 'Database Integrations' for Cloudflare Workers, a new feature that simplifies connecting Workers to popular HTTP-based databases like Neon, PlanetScale, and Supabase. It automates the discovery, OAuth2 authentication, and environment variable configuration process, significantly reducing developer friction. The post also mentions upcoming TCP socket support for Workers and the continued development of Cloudflare's own serverless SQL database, D1.

5/16/2023

This post announces a partnership with Snowflake, enabling users to query data stored in Cloudflare R2 object storage using Snowflake and load data from R2 into Snowflake. This extends Cloudflare's data infrastructure by providing a direct integration for advanced analytics and data warehousing, specifically addressing the challenge of data egress fees by leveraging R2's zero egress fee model.

5/15/2023

Introduced two new Cloudflare ChatGPT plugins: one for Cloudflare Radar, enabling natural language queries of Internet patterns via an enriched OpenAPI schema and itty-router-openapi, and another for Cloudflare Docs, using KV as a vector store for up-to-date documentation retrieval and Cron Triggers for index updates. This extends the capabilities of Cloudflare Radar by providing a new interface for data exploration and enhances developer experience by allowing natural language access to documentation.

5/12/2023

This post details the impact of internet disruptions in Pakistan, analyzing traffic data at national, administrative unit, and city levels. It observes shifts in traffic sources (mobile vs. desktop) and the increased use of Cloudflare's 1.1.1.1 resolver during content blocking. The analysis includes specific Autonomous System Numbers (ASNs) for mobile and fixed broadband providers, and metrics on traffic volume, bot vs. human traffic, and latency.

5/10/2023

This post details the analysis of internet traffic patterns in the UK during the coronation of King Charles III, correlating specific traffic spikes and dips with key moments of the ceremony and related events like the Coronation Big Lunch and Concert. It provides granular data on traffic changes (percentage drops/increases) by time, day, and region (England, Scotland, Wales, Northern Ireland), and analyzes the distribution of traffic from mobile devices. It also draws parallels with previous major events like Queen Elizabeth II's Platinum Jubilee and discusses the impact on other Commonwealth countries like Canada.

5/10/2023

This post introduces Object Lifecycle Management for Cloudflare R2, a new feature that allows users to define rules for automatically deleting objects after a specified period or aborting unfinished multipart uploads. This directly addresses the growing storage costs associated with applications that don't require indefinite data retention, providing a cost-management mechanism for R2 users.

5/2/2023

This post introduces Network Analytics v2, a fundamental redesign of the backend systems providing real-time network layer traffic visibility for Magic Transit and Spectrum customers. It addresses the limitations of the previous version by enabling the reporting of stateful mitigation systems (like flowtrackd) and adopting principles from traditional network observability (Netflow/sFlow). The new design allows individual software components to emit metadata-rich packet samples, creating a more flexible and future-proof observability system. Key technical challenges in modifying existing systems like xdpd and l4drop to emit these samples are discussed.

5/2/2023

This post details the analysis of Internet traffic patterns in Sudan during an armed conflict, showing significant drops in traffic, ISP outages, and shifts in messaging and social media platform usage, correlating these observations with real-world events and providing insights into the impact on connectivity. It leverages Cloudflare Radar data to illustrate these disruptions.

4/18/2023

Introduced the Aggregated Internet Measurement (AIM) initiative, a new open format for displaying Internet quality. AIM aims to bridge the gap between traditional speed test metrics (bandwidth, packet loss, latency, jitter) and end-user experience by correlating these metrics with specific use cases. AIM also aims to help diagnose the location of network problems (e.g., WiFi vs. ISP) and provides a publicly available repository for data analysis. The source code for AIM and its score calculations is now open-sourced.

4/12/2023

Introduced a new Network Analytics dashboard for Magic Transit and Spectrum customers, featuring a refactored network-layer data logging pipeline. The dashboard provides enhanced visibility into traffic behavior, firewall events, and DDoS attacks, with detailed mitigation analytics, geographical accuracy based on data center ingestion, and packet sampling. It enables faster DDoS response by allowing users to pattern traffic and create Magic Firewall rules directly from the interface, and supports report generation.

4/12/2023

This post details observed Internet disruptions across Q1 2023, categorizing them by cause (government-directed, cable cuts, power outages) and providing specific examples with traffic data and geographical context. It contributes to the ongoing effort to monitor and report on global internet resilience by providing a quarterly overview of observed disruptions.

3/15/2023

Introduced the Radar URL Scanner, a new free tool that compiles detailed technical reports for any given URL. The scanner leverages Cloudflare's Workers Browser Rendering API for headless scans and provides data across Security, Cookies, Network, Technology, DOM, and Performance categories. The post details the types of information available in each category and outlines future planned features such as API endpoints, private scans, and security recommendations.

3/3/2023

This post details how Cloudflare operates a large-scale Prometheus deployment (916 instances, 4.9 billion time series) for network monitoring. It explains the concepts of metrics, labels, cardinality, samples, and time series, and discusses the challenges of high cardinality and memory consumption in Prometheus. It also outlines the initial steps in the Prometheus lifecycle: HTTP scrape and TSDB storage.

2/23/2023

This post analyzes the impact of the war in Ukraine on internet trends, detailing traffic shifts, disruptions, and cyber attack patterns. It provides insights into network resilience, the effects of infrastructure damage, and the role of technologies like Starlink. The analysis leverages Cloudflare's network data to track changes in internet traffic volume and patterns across different regions and time periods, correlating them with geopolitical events and infrastructure attacks.

2/13/2023

This post details the analysis of Internet traffic trends during Super Bowl LVII using DNS name resolution data from Cloudflare's 1.1.1.1 resolver. It examines traffic spikes for specific advertisers (BlueMoon, LimitBreak, Temu, Dunkin'), analyzes traffic patterns for sports websites and team domains based on game events, and observes the impact of Rihanna's halftime show on messaging and social media traffic. It also explores traffic trends for video platforms and food delivery services during the event, highlighting how DNS data can be used to understand consumer behavior and advertising effectiveness.

2/3/2023

This post introduces Advanced HTTP Alerts, significantly enhancing Cloudflare's observability and analytics capabilities. Previously, HTTP alerts were generic. This update allows customers to customize alerts based on specific criteria like origin/edge response status codes, client IP addresses, and specific zones. This provides much finer-grained control for monitoring traffic anomalies and ensuring Service Level Objectives (SLOs) are met, directly contributing to the evolution of data infrastructure and analytics by providing more actionable insights from network traffic.

1/24/2023

This post details an advancement in Cloudflare's data infrastructure by introducing intelligent health checks for Kafka consumers. It addresses the challenges of ensuring application health in distributed systems like Kubernetes, particularly for Kafka consumers. The post explains the limitations of traditional health checks and proposes a new approach focused on message ingestion and offset commits to detect and automatically resolve unhealthy consumer states, thereby reducing incidents and manual intervention.

1/20/2023

This post details observed Internet disruptions globally during Q4 2022, including government-directed shutdowns in Cuba, Sudan, and Iran, and power outages in Bangladesh, Pakistan, and Kenya. It provides specific examples, dates, times, and accompanying traffic graphs illustrating the impact on Internet connectivity. The post also includes an analysis of the economic impact of these disruptions, particularly in Iran, citing statistics on business damage and income reduction.

1/9/2023

This post introduces Cloudflare's new product, Digital Experience Monitoring (DEM). DEM aims to provide organizations with visibility into end-user connectivity and performance issues, addressing the 'finger-pointing' problem between IT teams. Key features include Zero Trust Fleet Status, Synthetic Application Monitoring, and Network Path Visualization. This expands Cloudflare's data and analytics offerings by providing granular insights into the digital experience of users within Zero Trust deployments, leveraging Cloudflare's extensive network data.

12/22/2022

This post introduces the Cloudflare Radar 2022 Year in Review, presenting aggregated data on global Internet traffic growth (23% year-over-year), traffic trends influenced by global events (Olympics, Ukraine conflict, Hurricane Fiona), and category popularity (Technology, Business & Economy, Shopping & Auctions). It details methodology for traffic analysis and highlights regional variations in traffic distribution, such as in South Korea, Turkey, and Armenia. The post also discusses the impact of power outages on internet connectivity in Ukraine and Puerto Rico.

12/15/2022

This post details the launch of Radar Internet shutdown alerts and API access, enabling civil society organizations to track and document Internet disruptions. It highlights the collaboration with organizations like Access Now, Internet Society, and OONI, and introduces the Radar Outage Center (CROC).

11/25/2022

This post details internet traffic and e-commerce DNS trends observed during Thanksgiving 2022 in the US and Japan's Labor Thanksgiving Day. It highlights a 13% drop in US internet traffic during Thanksgiving dinner, followed by an increase on Black Friday. It also notes a dip in e-commerce DNS traffic during Thanksgiving dinner and an overall increase in e-commerce interest leading up to Black Friday. For Japan, it shows an increase in internet traffic during Labor Thanksgiving Day. The post also analyzes mobile device traffic trends during Thanksgiving and compares bot traffic patterns.

11/18/2022

This post introduces Workers Trace Events Logpush, a new capability that allows developers to send detailed logs from Cloudflare Workers to external object storage or analytics platforms. This extends existing debugging tools like `wrangler tail` by providing a historical record and enabling integration with popular observability tools, enhancing the overall observability of applications built on Cloudflare Workers.

11/18/2022

This post details how Cloudflare uses its own Workers Analytics Engine to instrument its SQL API. It explains the need for observability and product insights, the process of logging events from the SQL API into Analytics Engine, and how to query this data for insights into query response times and customer behavior. It highlights the use of Adaptive Bit Rate (ABR) for handling large datasets and the ability to identify and prioritize missing features based on aggregated error types and specific customer issues. The post also discusses future improvements to the Grafana plugin and data retention.

11/17/2022

This post details the engineering behind Radar 2.0, highlighting its architecture built on Cloudflare Pages with Functions, Remix for server-side rendering, and a Python/FastAPI backend API. It explains the use of Cloudflare Workers for the frontend API, transforming backend GraphQL data into a public REST API. Performance improvements through Early Hints and Lighthouse score enhancements are also discussed.

11/16/2022

This post introduces Cloudflare D1, a serverless SQL database service that integrates with Cloudflare Workers. It details how D1 enables developers to add dynamic, relational data to applications, specifically demonstrating its use for adding comments to a static blog site. The post covers setting up a D1 database, configuring bindings in wrangler.toml, executing SQL commands, and building API endpoints for creating and retrieving comments. It highlights D1's ability to augment existing applications and websites with dynamic data, bridging the gap between static site generators and fully dynamic applications.

11/15/2022

This post introduces the R2 Super Slurper, a new feature for Cloudflare R2 object storage. It addresses the challenge of migrating large amounts of data from S3-compatible storage to R2 by offering two migration strategies: a one-time 'giant slurp' for bulk imports and an 'automatic sip by sip' incremental migration that copies objects to R2 as they are requested. This simplifies data migration, reduces planning and staffing needs, and allows users to start saving on egress fees immediately with the incremental approach.

11/15/2022

Introduced the R2 Log Retrieval API, built using Cloudflare Workers and Durable Objects, to index and query millions of customer logs stored in R2. This system uses Durable Objects for maintaining forward-indexes of RayIDs per log batch and the Streams API to process compressed log data without OOM errors. The indexing process involves decompressing, decoding, splitting, and collecting RayIDs, which are then stored in Durable Object storage. Log retrieval optimizes searches by leveraging the timestamp encoded in RayIDs and batch names to narrow down the search space, streaming relevant batches from R2 and filtering for matching records.

11/15/2022

Introduced Cloudflare Logs Engine, a new product for storing and processing Cloudflare Logs within Cloudflare. This enables customers to retrieve logs by time range and unique identifier (RayID beta) from R2 storage. The product aims to separate storage and compute costs for affordability and integrates with analytics to narrow down search queries. Future plans include ingesting logs from all plan types, supporting arbitrary filtering, and cross-dataset queries.

11/14/2022

This post introduces Cloudflare Queues, a new open beta service that brings message queuing capabilities to the Workers Developer Platform. It enables developers to build more complex applications by decoupling components and buffering/batching calls to downstream services. The post details how to enroll in the beta, create a queue, and provides a practical example of using Queues with Workers and R2 for log aggregation, showcasing how Queues can improve efficiency and reduce costs.

11/3/2022

This post details the analysis of internet traffic patterns in Brazil during the 2022 Presidential elections using Cloudflare Radar data. It examines overall traffic trends, mobile device usage, and interest in election-related websites, candidate domains, and news organizations, correlating these with election events and official results. The analysis highlights how election days saw a decrease in daytime internet traffic, an increase in mobile usage, and significant spikes in DNS queries for election-specific content, with varying patterns for different categories of websites and candidates.

10/18/2022

This post details the observation and analysis of Internet disruptions during Q3 2022, categorizing them by cause (e.g., government-directed shutdowns) and geography, and leveraging Cloudflare Radar and internal tools to present traffic graphs and insights into these events. It highlights specific instances in Iraq, Cuba, Afghanistan, Sierra Leone, Somaliland, India, and Iran, providing traffic graphs and network-level details for each.

9/30/2022

Introduces Workers Analytics Engine, a new service for developers to store and analyze high-cardinality, time-series data generated by Cloudflare Workers. Highlights its use in powering Instant Logs by tracking request rates across sessions and data centers. Explains the underlying ABR technology for fast, interactive queries and the use of `_sample_interval` for reconstructing original event counts. Details improvements made for the open beta, including enhanced error messaging and comprehensive documentation. Mentions future plans for alerts, named fields, and a dashboard UX similar to Grafana.

9/30/2022

This post introduces Radar Domain Rankings, a new dataset based on aggregated 1.1.1.1 resolver data. It details the methodology for defining domain popularity (user population accessing a domain per unit of time, not just traffic volume) and the technical approach using machine learning models to predict domain ranks. It also explains the definition of a 'domain' for ranking purposes and the privacy considerations of using 1.1.1.1 data.

9/30/2022

This post introduces Cloudflare Radar 2.0, a significant redesign focused on improving the ease of finding and sharing Internet insights. Key technical contributions include a redesigned homepage with intuitive filtering and navigation, the introduction of 'quick bytes' for glanceable information, and enhanced content cards. The backend has been re-architected to facilitate the quick addition of new insights, with specific mention of email insights. Sharing capabilities have been expanded with the introduction of a public API for data access, and planned features for social media sharing and embeddable charts. The underlying technology stack has been modernized with GraphQL data endpoints, Cloudflare Pages and Workers for the website, server-side rendering using Remix, and a new reusable data visualization component system.

9/30/2022

Launched the Cloudflare Radar Outage Center (CROC) as part of Radar 2.0. CROC serves as an archive of Internet outage information, including location, ASN, type, scope, cause, start time, and end time. It is built on top of the Radar API, making outage data accessible programmatically. The CROC interface includes a global map and a table for viewing outage details. Future plans include increased automation of outage detection and monitoring of cloud platform providers.

9/28/2022

This post introduces a free, early-access version of Magic Network Monitoring (formerly Flow Based Monitoring), a self-serve network flow analytics tool. It allows users to visualize traffic, filter by packet characteristics, and set volumetric alerts. The post highlights its utility for network engineers and managers to understand network traffic, identify peak volumes, detect malicious activity like DDoS attacks, and integrate with Magic Transit for mitigation.

9/27/2022

This post introduces D1, Cloudflare's first SQL database designed to work seamlessly with Cloudflare Workers. It details the development journey, user experience with Wrangler CLI and dashboard, local development and testing capabilities, automatic backup and restore features, and a sneak peek into the implementation of JavaScript transactions using a new `db.transaction()` API within stored procedures executed via Workers. This significantly expands Cloudflare's data storage offerings beyond key-value stores, Durable Objects, and blob storage, enabling full-stack applications on Cloudflare's global network.

9/22/2022

Introduced filtering capabilities for Logpush jobs, allowing customers to specify criteria for log delivery to reduce data volume and cost. Added alerting for failing Logpush jobs, notifying users when jobs have been retrying and failing for 24 hours. Launched a GraphQL API endpoint (`logpushHealthAdaptiveGroups`) to provide analytics on Logpush job health, including metrics like bytes pushed, records pushed, and push status.

9/22/2022

This post details how Cloudflare Radar data was used to analyze and report on Internet disruptions in Iran during protests. It specifically shows the impact on traffic volumes for various Iranian ISPs (TCI, Iran Mobile Communications Company, RighTel, MTN Irancell) and regional/national traffic trends. It also analyzes DNS blocking by examining requests to Cloudflare's resolver for social media platforms (Instagram, WhatsApp) and the blocking of DNS-over-HTTPS/TLS (DoH/DoT) to Cloudflare's resolver. The analysis demonstrates the use of Cloudflare's network data to monitor and understand censorship and network interference.

9/22/2022

This post announces the general availability of Regional Services in India, Japan, and Australia, expanding the capability to control data processing locations to the Asia Pacific region. It reiterates the functionality of Regional Services, which allows customers to specify data centers that can decrypt and inspect traffic, thereby adhering to data localization regulations and contractual obligations. The post also explains the technical implementation of Regional Services, which works by forwarding raw, encrypted traffic to data centers within the customer-selected region for decryption and application of Layer 7 products.

9/21/2022

This post introduces the capability to store and retrieve Cloudflare logs directly on Cloudflare R2 object storage. Previously, logs could only be exported to third-party destinations. This new feature provides a cost-effective solution, eliminating egress fees and offering S3 API compatibility for existing tooling, thereby centralizing storage and retrieval for easier access and analysis.

9/15/2022

This post details how Cloudflare leverages its R2 object storage service to host apt/yum repositories for its cloudflared daemon. It explains the mechanics of package management systems like apt-get and yum, and outlines the process of creating a deb/rpm file, structuring the repository files, uploading them to R2, and serving them via a Cloudflare Worker. This demonstrates an innovative application of object storage for software distribution, enhancing reliability and reducing operational overhead.

9/2/2022

This post details the migration of Cloudflare's error logging pipeline from Elasticsearch to ClickHouse. It outlines the challenges faced with Elasticsearch, including mapping explosion, poor multi-tenancy support, operational overhead, and garbage collection issues, especially at Cloudflare's scale of 35-45 million HTTP requests per second with 500K-800K errors per second. The solution involves adopting ClickHouse, a column-oriented database, to improve query performance, reduce storage costs, and enable the storage of all error logs without sampling. Key ClickHouse features leveraged include its column-oriented storage for fast sequential scans, design for analytical workloads with many columns, sparse indexing, efficient LZ4 compression with configurable codecs (e.g., Double-Delta for DateTime, Gorilla for Float, LowCardinality for String), and linear scalability. An efficient inserter using Cap'n Proto messages and optimized batch sizes is also described.

8/26/2022

This post details the challenges and solutions for performance isolation in Cloudflare's multi-tenant Postgres clusters. It describes the problem of tenants contending for shared resources (CPU, memory, disk I/O, database connections) leading to throughput and latency degradation. The post outlines previous manual solutions like connection limits and introduces a new solution involving gateway concurrency throttling at the PgBouncer layer, including runtime commands and open-source contributions. It also explores future solutions like congestion avoidance inspired by TCP Vegas and tenant resource quotas to proactively manage performance and prevent resource starvation.

8/26/2022

This post contributes to the data infrastructure evolution by detailing improvements made to PgBouncer, a critical component for managing Postgres connection pooling. Specifically, it addresses an authentication bug that impacted HBA authentication and introduces new features for enforcing and dynamically adjusting per-user connection pool limits and per-connection pool limits at runtime. By open-sourcing these enhancements, Cloudflare not only improves its own database infrastructure but also benefits the broader open-source community, aligning with the theme of leveraging and contributing to foundational technologies.

7/26/2022

This post delves into the complex memory and window management of the TCP receiving side within the Linux kernel. It clarifies how Linux manages TCP receive buffers and windows, answering questions about the actual capacity of receive buffers and the speed at which they can be filled. The post explains the concepts of skmem_rb, recv-q, skmem_r, and the advertised window, detailing how settings like tcp_adv_win_scale influence performance. It highlights the critical issue of exceeding memory budgets, leading to packet drops and performance degradation, and discusses the roles of TCP Coalesce and TCP Collapse in mitigating these problems.

7/19/2022

This post details Cloudflare's extensive adoption and internal tooling development around Apache Kafka for inter-service communication. It highlights the use of Kafka to process over a trillion messages, the creation of internal tools like Messagebus-Client and a connector framework for easier adoption, the implementation of strict schemas using Protobuf for forward/backward compatibility, and the establishment of robust observability through automated metrics and alerts. It also provides a practical example of how this framework supports the Alert Notification System.

7/18/2022

This post details a debugging journey and fix for a crash in flowtrackd related to AF_XDP's lack of network namespace awareness. It highlights the use of AF_XDP for high-performance packet processing in DDoS mitigation, the challenges encountered with shared UMEM features across network namespaces, and the identification of a bug related to interface index numbers. The post also contributes to the understanding of low-level network infrastructure optimizations and their impact on security services.

7/5/2022

This post details the analysis of Internet disruptions observed by Cloudflare during Q2 2022, categorized by cause (physical events like fiber cuts, and intentional government shutdowns). It presents traffic graphs from Cloudflare Radar and internal tools to illustrate the impact of these events on specific networks (Comcast, Telkom SA, CANTV, AAE-1, SMW-5, CityWest) and countries (Syria, Sudan, Algeria). The analysis includes observations on asymmetric shutdowns, DNS retry floods, and the impact of content blocking versus nationwide shutdowns.

6/14/2022

This post details the use of Cloudflare Radar to track and analyze nationwide Internet shutdowns in Syria, Sudan, and Algeria during exam periods. It presents data showing traffic drops to zero, analyzes asymmetric inbound/outbound traffic patterns in Syria, and highlights the shift from full shutdowns to targeted website/application blocking in Algeria, supported by Network Error Logging (NEL) data. The post also references previous work on exam-related shutdowns in Sudan and the economic impact of such disruptions.

5/19/2022

This post details Cloudflare's use of Prometheus as its core monitoring system and introduces a new open-source tool developed to validate Prometheus alert rules. It explains the potential pitfalls of alert rule creation, such as typos or incorrect query logic, leading to silent failures where alerts don't fire when they should. The post elaborates on Prometheus querying basics (instant vs. range queries) and how these can lead to unexpected empty results, impacting alert reliability. The contribution lies in addressing the critical aspect of ensuring the accuracy and trustworthiness of the monitoring system itself, thereby improving the overall observability platform.

5/19/2022

This post details how Cloudflare Radar's DNS traffic analysis capabilities were used to observe the impact of the Eurovision 2022 song contest on internet traffic. It analyzes aggregate traffic to participating countries, traffic to fan websites, and traffic to national broadcaster websites, correlating spikes in DNS traffic with specific events during the semi-finals and final. The analysis highlights how DNS traffic to fan sites saw significant increases, peaking at 86x the usual traffic during the final. It also details specific moments of interest that caused traffic spikes on fan and broadcaster sites, such as song performances and the announcement of winners.

5/17/2022

Introduces Network Analytics Logs for Magic Transit, Magic Firewall, and Spectrum customers on the Enterprise plan. These logs provide packet samples of traffic dropped and passed by specific mitigation systems, enabling near real-time visibility into network traffic and DDoS attacks. The post details how to set up Logpush jobs to feed these logs into SIEM systems like Splunk, including API token creation, Splunk HEC setup, and the Logpush job configuration. It also highlights the cost-saving benefits of storing logs in R2.

5/16/2022

This post analyzes internet traffic patterns during Ramadan, correlating them with human behavior (fasting, pre-dawn meals, post-sunset meals). It uses Cloudflare Radar data to show country-specific shifts in traffic, particularly increases before sunrise and decreases after sunset. It also examines the overall daily traffic trends and post-Ramadan changes, highlighting how human events impact network usage.

5/12/2022

Introduced Workers Analytics Engine, a new serverless time-series analytics platform built on Cloudflare's existing analytics technology. It allows developers to collect and query telemetry data from Cloudflare Workers using a structured event log model with blobs (strings for grouping/filtering) and doubles (numbers for aggregation). Key features include low-latency reporting (seconds), unlimited cardinality of label values, fast queries across any timescale, and integration with SQL and GraphQL APIs for data querying. It also supports visualization with tools like Grafana. The initial implementation requires configuring bindings in wrangler.toml and using the `writeDataPoint` API within Workers. Future plans include named blobs/doubles, on-the-fly binding definition, and richer dashboard analytics.

5/11/2022

This post introduces the integration of Cloudflare's Logpush service with R2 object storage, allowing enterprise customers to store raw logs at a significantly reduced cost compared to traditional cloud object storage providers. It highlights the cost savings and introduces plans for enhanced log retrieval capabilities, including a Log Retrieval API beta.

5/10/2022

This post introduces a dedicated page on Cloudflare Radar for the Eurovision 2022 contest, providing insights into internet traffic trends related to the event. It showcases traffic aggregates in participating countries, DNS resolution data for video platforms and fan sites, social media trends, and the impact on national broadcasters' websites. The data is presented hour-by-hour and compared against a baseline of the previous week's average.

5/10/2022

This post announces the upcoming integration of Workers execution logs into Logpush for Enterprise customers. This new capability will provide unstructured console.log messages, exceptions, and request/response metadata, enabling granular debugging, performance analysis, and the exposure of logs to end-users for SaaS platforms built on Workers. It complements existing visibility tools like wrangler tail and Workers Analytics Engine.

4/20/2022

This post demonstrates the use of Cloudflare Radar to analyze traffic spikes to tax-related websites on US Tax Day 2022. It uses DNS name resolution data as a proxy for traffic and highlights the significant increase in requests to federal, state, and tax service websites on April 18, 2022. The analysis showcases Cloudflare Radar's ability to provide insights into internet traffic patterns during specific events.

4/11/2022

This post details the use of Cloudflare Radar data to analyze internet traffic patterns during the 2022 French Presidential election. It highlights observed trends such as decreased traffic during voting hours, increased traffic when election results were announced, and a significant rise in mobile device usage. The analysis also shows spikes in traffic to official election websites, news outlets, and TV/radio station websites around the time of result announcements.

4/5/2022

This post details a critical incident where a missing shell option ('pipefail') in a Kubernetes cron job led to corrupted configuration data in Quicksilver. This corruption caused the `dosd` service to fail, which in turn blocked requests in the Front Line service for up to five seconds per request, causing a global slowdown. The incident highlights the importance of robust error handling in data pipelines and the cascading effects of seemingly minor configuration oversights on critical infrastructure.

3/29/2022

This post details how Cloudflare Radar uses DNS name resolution data as a proxy for traffic to Internet services to analyze the impact of the 94th Academy Awards on internet traffic in the US. It provides specific examples of traffic increases for Twitter, TikTok, IMDb.com, ABC.com, Oscars.com, and Oscars.org, correlating these spikes with specific events during the ceremony. It also analyzes trends for movie news sites throughout the week leading up to and following the event. The post highlights the methodology used for baseline calculation and traffic analysis.

3/24/2022

This post introduces new Autonomous System (ASN) pages to Cloudflare Radar. These pages provide detailed traffic statistics, protocol usage, and security information for individual ASNs. They also display geographical traffic distribution and the volume of BGP announcements, offering insights into network changes and disruptions. The feature allows users to deep dive into any ASN by plugging its ASN into the Radar URL.

3/18/2022

This post introduces Cloudflare's future vision for observability, defining it as having three core components: monitoring, analytics, and forensics. It details how Cloudflare plans to enhance each of these areas: expanding notification types for monitoring, making the dashboard more customizable for analytics (especially for security policy validation), improving log access and storage (including on R2), and introducing tracing to provide end-to-end request lifecycle visibility. The post emphasizes the goal of providing a single pane of glass for all network activity and a unified experience across security, performance, and developer products.

3/14/2022

This post announces a new direct integration between Cloudflare's Logpush service and New Relic, a third-party observability platform. This integration aims to provide customers with enhanced end-to-end visibility by allowing them to correlate Cloudflare logs with data from their applications and origin servers within New Relic One. The partnership eliminates the need for middleware, leading to faster log delivery and reduced costs for mutual customers, and includes a quickstart guide for easy setup.

3/4/2022

This post details the analysis of internet traffic patterns in Ukraine from February 21, 2022, onwards, using Cloudflare's network data. It showcases how traffic data from various cities (Kyiv, Lviv, Uzhhorod, Ternopil, Rivne, Kharkiv, Sumy, Izyum, Donetsk, Mariupol, Osypenko, Irpin, Bucha, Enerhodar, Severodonetsk) can be used to infer real-world events such as population displacement and the impact of cyberattacks. It also analyzes cyberattack traffic, including DDoS attacks and layer 7 attacks against .ua domains, demonstrating Cloudflare's mitigation efforts and the types of attacks observed.

2/14/2022

This post details the use of Cloudflare Radar's DNS name resolution data to analyze Internet traffic trends during Super Bowl LVI. It quantifies traffic spikes to various website categories (food delivery, sports, video, social media, messaging) and specific company websites (teams, advertisers) in response to game events and commercials. The analysis methodology involves establishing a baseline traffic level and measuring deviations, highlighting the impact of advertising on website visitation.

2/7/2022

This post introduces the ability to consume Cloudflare Instant Logs via WebSockets and analyze them in real-time on the command line using tools like Websocat and Angle Grinder. It details how to create an Instant Logs session, specify desired log fields (e.g., ClientIP, FirewallMatchesActions), set sample rates, and apply filters based on request path. A practical example demonstrates how to verify a newly deployed firewall rule by analyzing logs for requests to a specific path and filtering by country and firewall actions.

2/4/2022

This post delves into the intricacies of `io_uring`, a high-performance asynchronous I/O API in Linux, specifically focusing on its worker pool management. It clarifies how `io_uring` classifies I/O requests into bounded and unbounded work, and how it manages thread creation for each category. The post details how to monitor and control the unbounded worker pool size, which is crucial for network I/O operations that are central to Cloudflare's infrastructure. It provides practical examples using in-kernel tracing to observe `io_uring`'s behavior and offers solutions for developers to effectively utilize this powerful I/O runtime.

1/26/2022

This post quantifies the growth of API traffic on Cloudflare's network in 2021, showing it now represents 54% of total requests and is growing twice as fast as traditional web traffic. It details API growth by industry, highlighting significant increases in Banking, Retail, and Financial Services. The post also analyzes API characteristics, noting POST and GET as the dominant methods (98% of requests) and JSON as the primary payload encoding (97% of requests). It introduces Cloudflare's API Shield products (API Discovery, Schema Validation, mTLS, API Abuse Detection) as tools to protect against API-specific threats.

1/5/2022

This post details Cloudflare Radar's observation and analysis of the nationwide internet shutdown in Kazakhstan in January 2022. It uses Radar data to show traffic drops across various ASNs, including mobile services, and correlates these with BGP updates. The post also tracks intermittent restorations of internet services, linking them to public announcements by the Kazakh President. It highlights the impact on businesses and Bitcoin mining, and compares the event to other internet shutdowns.

12/23/2021

This post introduces Cloudflare Radar's 2021 Year In Review, detailing trends in Internet traffic, mobile vs. desktop usage, and attack distributions throughout the year. It highlights the impact of the COVID-19 pandemic on internet usage, with lockdowns and restrictions influencing traffic patterns globally. Specific country-level data is provided, along with an analysis of the 'Thanksgiving effect' on mobile traffic in the US. The post also discusses how specific events like the Colonial Pipeline cyberattack influenced attack peaks.

12/20/2021

This post details the evolution of Cloudflare Radar's capabilities in analyzing and presenting internet traffic data. It highlights the platform's ability to track domain popularity, comparing trends from 2020 to 2021, and specifically focuses on the rise of TikTok, the performance of e-commerce giants like Amazon, and the dynamics of social media platforms. The post also delves into specific events that influenced traffic patterns, such as the Facebook outage and viral YouTube content, and discusses the methodology behind Cloudflare's domain ranking system, which relies on data from its public DNS resolver (1.1.1.1).

12/16/2021

This post details the engineering behind Cloudflare's Crawler Hints product, which aims to reduce the environmental impact of web searches by informing search indexers about content changes. It explains how cache miss data from Cloudflare's global network is processed, utilizing Kafka for message buffering and Redis for aggregation and deduplication. The post also covers the integration with search partners like Yandex and Bing through the IndexNow API and highlights the importance of customer opt-in, monitoring, and system resilience in rolling out such a large-scale feature.

12/14/2021

This post addresses a critical security vulnerability (Log4j) by introducing a log sanitization feature for Cloudflare Logs. Customers can now opt-in to automatically redact potentially exploitable tokens (`${`) within their logs via Logpush jobs, protecting them from remote code execution attempts when consuming logs with vulnerable software. This enhances the security and integrity of the data infrastructure analytics pipeline.

12/11/2021

This post details significant enhancements to Cloudflare's Notification center. It highlights the importance of timely notifications for customers to be aware of issues impacting their internet properties. The post announces the shift from primarily email-based notifications to a greater emphasis on webhooks, with expanded support for platforms like DataDog, Discord, OpsGenie, and Splunk. Crucially, it introduces Notification History, allowing users to review past notifications for up to 90 days (depending on plan), addressing the issue of lost or missed alerts.

12/7/2021

This post introduces the ability to store Cloudflare logs directly on Cloudflare R2 storage. This provides a cost-effective, long-term solution for retaining raw log data from all Cloudflare products, enabling deeper analysis and forensics capabilities directly on Cloudflare. It addresses customer requests for low-cost, low-effort, long-term log storage accessible for any product.

11/30/2021

This post details the analysis of e-commerce traffic trends during November, highlighting Cyber Monday as the global winner and providing country-specific breakdowns for the US, UK, Germany, France, India, and Japan. It also analyzes the increase in mobile traffic percentage towards the end of November and during Black Friday. The analysis leverages Cloudflare Radar's capabilities to visualize and interpret these traffic patterns.

11/29/2021

Introduced Radar Maps with geographical distribution of application-level attacks (origin and target) and Sankey diagrams for attack flows. These visualizations are built using Cloudflare Workers and Workers KV.

11/26/2021

This post demonstrates the use of Cloudflare Radar to analyze internet traffic patterns during the US Thanksgiving holiday. It highlights how Radar can show dips in e-commerce traffic on Thanksgiving Day, shifts in peak traffic times, and an increase in mobile device usage. The analysis uses visualizations of internet traffic volume and mobile traffic percentage to illustrate these trends.

11/22/2021

This post details the restoration of internet traffic in Sudan after a 25-day shutdown, using Cloudflare Radar data to illustrate the recovery patterns. It highlights the role of specific ISPs (Mobitel, MTN, Sudatel) in the restoration and notes the impact on mobile traffic, which is dominant in Sudan. The post also contextualizes the event by referencing previous internet shutdowns in the country and emphasizes the ongoing utility of Cloudflare Radar for monitoring global internet traffic and disruptions.

11/5/2021

This post analyzes global internet traffic trends, correlating shifts in mobile vs. desktop usage with the back-to-school season in the Northern Hemisphere. It presents data visualizations showing these trends across Europe, Spain, Portugal, Italy, the Netherlands, Japan, the US, Canada, China, Russia, Cyprus, and Nigeria. The analysis highlights how factors like school calendars, public holidays, and COVID-19 lockdowns influence internet access patterns. It also notes a general increase in overall internet traffic in September compared to August.

11/1/2021

This post details the use of Cloudflare Radar to monitor and report on a country-wide internet outage in Sudan. It highlights the flatlining of internet traffic, the minimal blips of activity, and provides historical context of previous internet shutdowns in the region. The post demonstrates Radar's capability to track and visualize internet access disruptions at a national level.

10/25/2021

This post details the use of Cloudflare Radar to track and visualize Internet traffic disruptions in Sudan during a political event. It presents data showing a significant drop in overall Internet traffic, HTTP requests, and traffic across all major ASNs. The post also highlights the impact on both mobile and desktop traffic and notes a change in Layer 3&4 DDoS attack patterns. The data is presented through screenshots of Radar's interface, demonstrating its capability to monitor global and country-specific Internet health.

10/15/2021

This post introduces a novel method for passively measuring and characterizing CDN latencies. It addresses the limitations of active measurements (like RIPE Atlas and RUM services) due to probe distribution, data validation issues, and privacy concerns. The post details a two-step process: first, predicting anycast catchments by compiling network footprints (ISPs, data centers, IXPs, BGP data) to infer which CDN data center a client request will reach; and second, predicting CDN path latencies by leveraging Cloudflare's own passive RTT measurements as a proxy. This contributes a new analytical capability to understand and compare CDN performance without direct, active probing of competitors.

10/15/2021

This post introduces a new capability within Cloudflare's data analytics to detect multi-user IP addresses. It explains the challenges posed by shared IP addresses due to NAT and CG-NAT, and details the methods developed to identify these multi-user IP prefixes by analyzing distinct source ports and user agents. This enhances Cloudflare's security tools by providing more granular traffic identification.

10/9/2021

This post introduces analysis of mobile vs. desktop traffic trends across different countries and regions, highlighting global disparities and regional preferences. It leverages Cloudflare Radar data to identify countries with the highest and lowest proportions of mobile internet traffic, providing insights into regional internet usage patterns.

9/30/2021

This post demonstrates the use of Cloudflare Radar's Global Popularity Ranking Trend to track and visualize increased internet traffic to Spanish media outlets (El País, El Mundo, RTVE) following the La Palma volcanic eruption. It highlights how Radar can be used to identify and analyze real-world event impacts on internet trends.

9/28/2021

This post announces Cloudflare R2 Storage, a new object storage service. R2 offers "Really Requestable" and "Ridiculously Reliable" object storage with zero egress fees, full S3 API compatibility, and competitive pricing for storage and operations. It integrates deeply with Cloudflare Workers for dynamic data transformation and pipeline building, and is designed for high durability and availability. R2 aims to democratize data storage by removing egress costs, making it ideal for CDN assets, media files, IoT data, and applications requiring large-scale, cost-effective storage.

9/16/2021

This post introduces 'Last Mile Insights', a new feature that provides customers with visibility into connection failures occurring between their end-users and Cloudflare properties. It explains the concept of the 'Last Mile' in internet connectivity, details how Last Mile Insights works using Network Error Logging (NEL) and machine learning, and presents a case study with Canva to illustrate its value in diagnosing and resolving ISP-related connectivity issues. The post also reiterates Cloudflare's commitment to global network expansion to minimize Last Mile impact.

9/14/2021

This post introduces 'Instant Logs,' a new real-time log access feature built on Cloudflare Workers and Durable Objects. It addresses the need for immediate visibility into network events, complementing the existing Logpush product by providing sub-second latency through edge-based processing, filtering, and sampling (Reservoir Sampling) to handle varying traffic volumes.

9/14/2021

Introduced Live-updating Analytics for Pro, Business, and Enterprise customers, providing real-time visibility into data as it arrives. Launched Instant Logs for Enterprise customers, enabling near real-time viewing of HTTP request logs in the dashboard with an average delay of two seconds. This significantly reduces the 'glass-to-glass' time for log data, enabling faster response to security threats and operational issues.

9/8/2021

This post introduces 'Custom Hostname Analytics,' a feature that allows SaaS providers to offer their end-customers visibility into traffic analytics for their specific domains hosted on the SaaS platform. This extends Cloudflare's existing analytics capabilities to provide granular insights for individual customer hostnames, enabling SaaS providers to better understand customer usage, billing, and infrastructure scaling, and to demonstrate the value of Cloudflare's performance and security benefits to their own clients.

8/24/2021

Introduced integrated logging and exception tracking for Cloudflare Workers, accessible directly from the dashboard and via the `wrangler tail` CLI. This feature eliminates the need for manual logging setup by developers, automatically capturing `console.log` output and exceptions. The dashboard provides a viewable stream of logs and exceptions, with filtering by event status and type. The `wrangler tail` command offers advanced filtering by IP address, status, HTTP method, sampling rate, and generic search queries, with options for pretty and JSON output formats. This enhancement aims to simplify debugging and improve the understanding of Worker execution.

7/29/2021

This post introduces Radar Alerts, a new feature built on Cloudflare's existing Radar platform. Radar Alerts leverages Cloudflare's global network visibility to detect and notify partner organizations about significant drops in internet traffic, which are often indicative of internet shutdowns. The system includes an additional validation layer and a notification system that sends alerts through various channels. This enhances the ability of civil society groups to track, document, and hold institutions accountable for internet disruptions.

7/26/2021

This post introduces Project Pangea, an initiative to reduce Internet inequality by providing fast, reliable transit to ISPs in underserved communities. It presents an analysis of Internet performance (bandwidth and latency) across eight countries (US, Brazil, UK, Germany, France, South Africa, Japan, Australia) to highlight disparities and inform infrastructure investment. The analysis confirms that Internet access inequality exists globally, even within developed nations, and that averages can mask significant local performance issues. Cloudflare is committed to making more of this data and analysis available.

6/22/2021

Introduced new datasets for Cloudflare Logs: Firewall Events and Network Error Logging (NEL) Reports. Expanded Logpush destinations to include any S3-compatible API. Enabled direct integrations for pushing logs to Microsoft Azure Sentinel, Splunk, Sumo Logic, and Datadog via the UI. Redesigned the Logpush UI for selecting datasets and destinations.

6/22/2021

This post details the use of Cloudflare Radar to track Internet traffic patterns in Sudan, specifically identifying and analyzing Internet shutdowns implemented by the government during exam periods. It highlights the ability to observe traffic drops, differentiate between mobile and desktop access during these shutdowns, and provides a direct link to the country-specific Radar page for real-time monitoring. The post also mentions the potential for future shutdowns and the use of Twitter for assistance in tracking them.

6/3/2021

This post details the use of Cloudflare Radar to track and visualize internet shutdowns in Syria during exam periods. It highlights how Radar's country-specific pages and traffic data can be used to observe significant drops in internet usage corresponding to scheduled outages, providing concrete evidence of these events.

5/20/2021

This post details the implementation of a custom Prometheus exporter by Labyrinth Labs that pulls data from Cloudflare's GraphQL Analytics API. The exporter transforms Cloudflare analytics data (requests, bandwidth, cache utilization, threats, SSL usage, HTTP response codes, traffic origin by country/location, and data center performance) into Prometheus metrics. It includes design considerations (Go, API SDK, Goroutines for parallel requests), deployment instructions (Docker image, Helm chart for Kubernetes), and visualization guidance using Grafana dashboards. Future work includes integrating more analytics data like firewall, DoS, and network analytics.

4/16/2021

This post announces Cloudflare's strategic partnerships with Macrometa and Fauna to enhance state management at the edge for Cloudflare Workers. It introduces Macrometa as a globally replicated NoSQL database with integrated search, pub/sub, and stream processing capabilities, and Fauna as a global transactional database delivered as a secure, cloud API. The post highlights how these partnerships enable developers to build stateful applications with low-latency data access, querying, and strong consistency, complementing existing Workers KV and Durable Objects by addressing use cases like complex data querying and integration with existing systems of record. Case studies for a bookstore demo app with Macrometa and the MeetKai AI assistant with Fauna are provided to showcase the benefits.

3/19/2021

This post announces the integration of Cloudflare's privacy-first web analytics into WordPress.com, allowing publishers to collect usage data and gain insights about their visitors. It also highlights the popularity and success of Automatic Platform Optimization (APO) for WordPress, which improves TTFB, and details how users can enable these features within the WordPress.com dashboard. This partnership signifies a step towards a better, more private internet.

3/15/2021

This post introduces several enhancements to Cloudflare's Web Analytics platform. It adds support for measuring multiple websites per account (up to 10), with the ability to view combined analytics across all sites using tags. Crucially, it introduces support for Single-Page Applications (SPAs) by tracking changes to the History API (pushState and onpopstate events) to capture route changes. Additionally, Core Web Vitals are now available within Web Analytics for all customers, providing insights into website performance.

3/12/2021

This post details the evolution and scaling of Cloudflare's Anomaly Detection platform, a key component of its Bot Management system. It highlights the platform's technical implementation using HBOS, ClickHouse, and Redis, and chronicles its journey from a single monolithic service struggling with 10k RPS to a microservices-based architecture handling nearly 500k RPS. Key contributions include optimizations in Redis (PFMERGE, recency register, key encoding) and the strategic shift to microservices to reduce load on shared dependencies like ClickHouse and improve independent scalability of pipeline components.

3/4/2021

This post delves into the intricacies of the Linux kernel's conntrack module and its interaction with the Netfilter framework. It explains why conntrack doesn't log dropped SYN packets by default, how to enable its hooks, and uses BPF tracing to demonstrate the flow of packets through Netfilter. This contributes to the understanding of data infrastructure by providing a deep dive into a fundamental network component that impacts traffic analysis and observability.

2/8/2021

This post details the use of Cloudflare Radar's DNS name resolution data to analyze Internet traffic patterns during Super Bowl LV. It specifically examines the traffic spikes to advertisers' websites, social media usage, food delivery services, and football-related websites, correlating these with specific game events and commercials. The analysis highlights how DNS data can estimate website interest and provides insights into the impact of Super Bowl commercials on advertiser website traffic.

1/27/2021

This post introduces Cloudflare's Provisioning-as-a-Service (PraaS) platform, built using Apache Airflow, to automate the complex process of data center expansions and server provisioning. It details how manual Standard Operating Procedures (SOPs) were transformed into API-driven tasks within Airflow DAGs, integrating with tools like Salt, Prometheus, and JIRA. The post highlights features like failure handling, logging, notifications, Jinja templating, sensors for preconditions and human intervention, branching, multi-DAGs, and strategies for creating scalable DAGs to manage over 200 data centers, resulting in a 90% reduction in manual operational tasks.

1/15/2021

This post details the analysis of the January 2021 internet shutdown in Uganda using Cloudflare Radar data. It demonstrates how traffic patterns, application-level attacks, and BGP routing statistics from Cloudflare's network and the Uganda Internet eXchange point were used to monitor and report on the outage and its subsequent restoration. The post highlights the ability to visualize significant drops and returns to near-zero internet traffic, providing concrete data on the impact of government-ordered internet suspensions.

1/14/2021

This post introduces SOAR (Simulation for Observability, Reliability, and Security), a new system designed to address the challenges of testing complex, homogeneous edge environments. SOAR creates dedicated data centers for running simulations with synthesized customer traffic and applications, mirroring production software stacks. It enhances isolation and coordination between simulations, improves engineer-friendliness through an internal coordinator service, and allows for more complex, interactive simulations, particularly for products like Magic Transit, ensuring reliability and security before production deployment.

1/12/2021

This post introduces Cloudflare Radar and its Year In Review feature, which provides public access to Internet use and abuse trends. It details how Radar tracks traffic patterns, geographic access shifts (e.g., from urban centers to residential areas during lockdowns), and changes in user activity categories (e.g., increased e-commerce, decreased travel). The post uses interactive maps and charts to visualize these trends for 2020, highlighting the impact of the SARS-Cov-2 pandemic on Internet usage.

1/6/2021

This post analyzes the impact of the Christmas Day bombing in Nashville on internet traffic flow, specifically focusing on the disruption to AT&T services and its effect on Cloudflare traffic. It presents traffic flow data before, during, and after the outage, illustrating the duration and recovery of services. The analysis demonstrates the use of traffic flow data to understand real-world network disruptions.

12/18/2020

This post details the engineering challenges and solutions for computing Euclidean distance on high-dimensional data (144 dimensions) for an image matching algorithm used in Cloudflare's CSAM scanning tool. It explores naive brute-force approaches, the limitations of SIMD (AVX2) due to memory bandwidth, and the failure of space partitioning algorithms like VP-trees due to the curse of dimensionality. The post highlights a breakthrough with a 'short distance' variation that computes only a subset of dimensions initially, and further optimization by leveraging the application's characteristic of expecting few matches (proving non-existence of neighbors within a threshold) to achieve significant performance gains.

12/9/2020

This post announces the public availability of Cloudflare Web Analytics, allowing users to integrate it into any website via a JavaScript snippet. This expands the analytics offering beyond existing Cloudflare customers who previously relied on DNS changes. The post details the privacy-first approach, emphasizing no user tracking, no cookies, and no fingerprinting. It explains the core metrics (page views, visits) and features like global filters, zoom, and group-by functionality. The technical implementation leverages client-side tracking, similar to other popular analytics tools, contrasting with Cloudflare's traditional edge-based analytics.

11/27/2020

This post details a significant incident involving a Byzantine fault in the etcd cluster due to a partial switch failure, which cascaded into database primary promotion issues and impacted API and dashboard availability. It highlights the challenges of handling degraded states in redundant systems, the limitations of RAFT in Byzantine scenarios, and the subsequent improvements made to database cluster management, user session handling, and auto-remediation configuration. It also touches upon the ongoing research into Byzantine Fault Tolerance (BFT) for production systems.

11/25/2020

This post details the production deployment of Quicksilver, Cloudflare's new distributed data store for configuration distribution, which replaces the older Kyoto Tycoon. It describes the "bridge" strategy for parallel operation, the gradual rollout process, and the technical challenges encountered, such as replication saturation due to bootstrapping, I/O errors from aging SSDs, write amplification issues with LMDB, and the need for more dynamic topology management. The post highlights significant performance improvements observed after migrating critical services like FL to Quicksilver.

11/5/2020

Introduced a capacity estimation framework for ClickHouse clusters to automate disk space forecasting and proactive hardware procurement. This framework leverages historical metrics from system.parts, filters for unreplicated data to account for topology changes, and uses time-series forecasting models like Prophet for predictions. The results are visualized in Grafana, providing insights into real and predicted disk usage.

10/27/2020

This post details the implementation of the `/proc/[pid]/mem` file within gVisor's virtual file system. This was a crucial step to enable proper stack unwinding and debugging for processes running inside the sandbox, which previously resulted in unhelpful stack traces. The post dives into the Linux kernel's implementation of `/proc/[pid]/mem`, including access checks and read operations, and explains how these concepts were adapted for gVisor's sandboxing environment. This directly contributes to Cloudflare's data infrastructure and analytics by enhancing the diagnostic capabilities within their sandboxed environments.

10/21/2020

Introduced a special "Election 2020" Radar dashboard to publicly share data on traffic to elections-related sites and observed cyberattacks, providing transparency into the security landscape surrounding the election. This dashboard is updated continuously with information from internal monitoring systems.

9/30/2020

This post introduces Cloudflare Radar, a new public-facing service that exposes aggregated data on Internet traffic patterns, domain popularity, and cyberattack activity. It comprises three key components: Radar Internet Insights (news, trend data, domain popularity, attack activity, technology trends), Radar Domain Insights (detailed information on individual domains including popularity, security, and content categories), and Radar IP Insights (network and geographic information for IP addresses). This launch democratizes access to insights previously only available internally at Cloudflare.

9/29/2020

Introduced a new, privacy-first web analytics service accessible to all users via a JavaScript beacon, complementing existing edge analytics. This service defines 'visits' as a privacy-friendly measure of interaction, avoiding individual user tracking, client-side state (cookies, localStorage), and IP address fingerprinting. It offers features like drag-to-zoom and grouping for time-series charts and leverages ABR analytics technology for fast data serving. Plans include bot detection and removal for customers of Bot Management.

9/29/2020

Introduced Adaptive Bit Rate (ABR) analytics for Cloudflare's data infrastructure. ABR allows analytics queries to be calculated at resolutions matching the query's needs, optimizing response times and resource usage by storing data at multiple resolutions. This technique improves resilience and scalability by adapting to system load and tail latency, ensuring consistent dashboard performance. The post also details the application of ABR to aggregations like topK and count-distinct, using sampling techniques for efficient estimation.

9/29/2020

This post introduces the integration of Web Vitals (Largest Contentful Paint, First Input Delay, Cumulative Layout Shift) into Cloudflare's Browser Insights product. It explains the importance of Web Vitals for understanding user experience beyond traditional metrics like TTFB. The post highlights the benefits of using Cloudflare's Browser Insights for RUM data, including continuous updates, support for all browsers (not just Chromium), and its free availability. It details how to enable Browser Insights via the Speed tab in the dashboard and mentions the underlying JavaScript beacon mechanism.

6/15/2020

This post introduces the application of data science and machine learning to improve customer support and identify attack traffic. It details three specific techniques: improving language classification by considering multiple signals beyond off-the-shelf algorithms, optimizing fuzzy string matching for error message extraction using Cosine similarity for better performance, and developing a multi-dimensional data analysis approach with unsupervised clustering for identifying complex attack patterns like credential stuffing.

6/12/2020

This post introduces Standalone Health Check Analytics, a new feature that provides detailed logs and visualizations for diagnosing origin server issues. It enhances existing health check capabilities by offering error breakdown, availability graphs, response code mismatch details, and a request waterfall view to analyze TCP connection, TLS handshake, and time to first byte. It also highlights the global nature of health checks, showing regional latency variations.

6/5/2020

This post details the adoption of HashiCorp Nomad for managing 'management services' in Cloudflare's edge data centers. This directly contributes to the reliability and operational efficiency of the data infrastructure that underpins analytics capabilities by ensuring critical services remain available and efficiently resourced, even in distributed environments. It highlights how Nomad's dynamic task scheduling addresses challenges in managing services that are not customer-facing but essential for data center operations, thereby indirectly supporting the overall data infrastructure and analytics ecosystem.

5/29/2020

Introduced the ability to export logs from Cloudflare Gateway using the Cloudflare Logpush Service. This allows organizations to analyze and audit tens or hundreds of millions of DNS queries and security decisions by exporting data to third-party storage destinations or SIEM tools. The feature builds on the existing Logpush Service and provides configuration options for fields and storage destinations within the Cloudflare for Teams UI.

5/26/2020

Introduced speed.cloudflare.com, a network performance testing tool that measures download, upload, latency, and jitter. The tool provides transparent methodology, allows users to download raw measurements, and runs entirely on Cloudflare Workers at the edge. The backend code is open-sourced on GitHub.

5/13/2020

This post details the integration of Backblaze B2's S3-compatible API with Cloudflare Workers. It provides a practical example of how to use Cloudflare Workers and the aws4fetch library to sign requests to Backblaze B2, enabling direct serving of content from Backblaze B2 through Cloudflare's network. It also highlights the benefits of the Bandwidth Alliance in reducing first-mile bandwidth costs.

5/8/2020

This post details the integration and testing of CUBIC and HyStart++ congestion control algorithms within Cloudflare's quiche (QUIC) implementation. It explains the concepts of QUIC congestion control, Reno, CUBIC, and HyStart++, and presents lab test results demonstrating that CUBIC and HyStart++ improve performance and significantly reduce packet loss compared to Reno, especially under packet loss conditions. This contributes to the ongoing optimization of Cloudflare's network transport layer for better performance and reliability.

4/24/2020

This post introduces the ability to stream Firewall Events directly to SIEMs using Cloudflare Logs and Logpush jobs. It details the process of setting up this integration using Terraform and Sumo Logic, including prerequisites, Sumo Logic collector and HTTP source creation, and Cloudflare API token generation. The key technical contribution is enabling customers to send only security-relevant events to their logging platforms, reducing data volume and improving the speed of event delivery (within 60 seconds).

4/16/2020

This post details a significant outage of the Cloudflare Dashboard and API due to a physical disconnection of critical fiber optic cables during planned maintenance. It highlights the importance of robust internal data infrastructure and control plane availability, the challenges of disaster recovery and failback, and the need for improved documentation, labeling, and process design to prevent single points of failure in physical infrastructure.

4/7/2020

This post details the refactoring and revamp of 'Crossbow', an internal tool used by Cloudflare's Technical Support Engineers for network diagnostics. It highlights the process of deprecating the UI to focus on the CLI, rearchitecting the pub/sub pipeline for better stability and security, and integrating JWT authentication with Cloudflare Access. The post also emphasizes the importance of risk management, feedback loops, and test-driven development in improving internal tooling, leading to a significant increase in usage and the consolidation of other diagnostic tools into Crossbow.

3/30/2020

This post introduces Quicksilver, a new system designed to replace Kyoto Tycoon for configuration distribution at internet scale. It details the limitations of the previous system, Kyoto Tycoon, particularly its performance degradation under heavy write loads due to exclusive write locks and issues with data synchronization and corruption. Quicksilver is presented as the solution to these problems, enabling faster and more reliable distribution of configuration changes across Cloudflare's global network.

3/16/2020

Introduced Network Analytics dashboard for Magic Transit and BYOIP customers, providing packet-layer visibility into network and transport-layer traffic patterns and DDoS attacks. This extends the existing analytics platform to L3/L4, complementing the previous L7-focused dashboards. The Network Analytics dashboard offers near real-time visibility into traffic and attacks blocked at the edge, with data accessible via GraphQL. Data storage architecture provides one year of insights, with sampled IP flow logs and summarized attack logs. The post details customer use cases for incident response and reporting, highlighting key metrics for each.

3/2/2020

This post introduces the use of Bloom filters as a probabilistic data structure for efficient large-scale data deduplication, specifically in the context of analyzing IP spoofing data. It details the development of a custom tool 'mmuniq-bloom' and uses profiling to identify performance bottlenecks related to random memory access and cache misses, demonstrating the practical challenges and trade-offs of using Bloom filters for massive datasets.

2/27/2020

This post details Cloudflare's in-depth performance tuning efforts for their AMD EPYC 7642 processors in their Gen X servers. It explores the nuances of Thermal Design Power (TDP) and dynamic power, the impact of determinism modes (Performance vs. Power), and the benefits of Configurable TDP (cTDP). The post quantifies the gains achieved, including an additional 6% performance throughput from out-of-the-box tuning, with further improvements of 2% and 4% from power determinism and cTDP adjustments respectively. It also touches upon the Nodes Per Socket (NPS) configuration, noting no significant performance deltas observed.

2/26/2020

This post contributes to the data infrastructure and analytics thread by detailing an in-depth hardware performance analysis. It specifically evaluates the impact of cache locality on Cloudflare's workloads by comparing AMD 2nd Gen EPYC processors with Intel Xeon processors. The post highlights how advancements in CPU architecture, particularly larger L3 caches, can significantly improve request processing throughput and reduce cache miss rates, leading to more efficient server performance.

2/25/2020

This post details Cloudflare's strategic decision to adopt AMD EPYC 7642 processors for its 10th-generation edge servers, replacing Intel Xeon Platinum 6162. The primary driver for this change is the pursuit of higher 'Requests per Watt,' a key metric for operational efficiency and cost management. The post provides a detailed comparison of the new AMD CPUs against the previous Intel generation, highlighting improvements in core density, power efficiency (TDP per core), memory channels, and PCIe lanes. Extensive benchmarking across cryptography, compression, and Go-specific operations demonstrates significant performance gains with the AMD EPYC processors, reinforcing their suitability for Cloudflare's compute-intensive workload.

12/12/2019

This post introduces the new GraphQL Analytics API and demonstrates its use in building Firewall Analytics. It details how the API enables querying firewall event aggregates (`firewallEventsAdaptiveGroups`) and individual events (`firewallEventsAdaptive`) using a schema-driven approach. Specific examples are provided for visualizing time-series data of firewall events by hour and action, identifying top N attributes (client IP, user agent) contributing to events, and retrieving raw event details for false positive analysis. The post also explains GraphQL query structure, schema exploration, and the use of filters and dimensions for data aggregation.

12/12/2019

Introduced a new GraphQL Analytics API as a unified endpoint for accessing Cloudflare metrics and logs, replacing the previous multi-API approach. This API allows users to select specific data points and aggregate them across domains, facilitating data exploration and dashboard creation. The post details the API's introspection capabilities and provides examples of queries for aggregated requests, zone-specific traffic comparison, and correlation of error responses with firewall events. The Zone Analytics API is deprecated in favor of this new GraphQL API.

12/10/2019

This post introduces Load Balancing Analytics, a new feature that provides detailed insights into traffic steering decisions, origin health, and latency for Cloudflare Load Balancing. It highlights the traffic flow overview, a latency map, and the underlying GraphQL Analytics API, enabling customers to optimize their infrastructure, reduce costs, and increase application availability.

12/9/2019

Introduced Firewall Analytics for Business and Pro plans, highlighting new metrics like solved captcha rate and customizable reports. Launched Load Balancing Analytics showing traffic flows by load balancer, pool, origin, and region. Announced new origin monitoring tools (active and passive) with automatic traffic rerouting. Made the underlying API for product analytics generally available for custom dashboards and data exploration. Released Account Analytics (beta) for aggregated domain information.

8/22/2019

This post introduces a completely overhauled Firewall Event log for Free, Pro, and Business customers. Key improvements include a smoother user experience without modals, expanded detail views for each event, an 'Additional matches' count to clarify rules and services triggered for a single request, and powerful freeform search capabilities across all visible fields within a Firewall Event. It also adds the ability to scope searches to specific date and time windows (24 hours for Free/Pro, 72 hours for Business) within the last two weeks.

7/10/2019

This post introduces the application of coverage-guided fuzzing, specifically using AFL and KCOV, to the Linux Kernel's netlink machinery. It details how to set up and run this process within a KVM virtualization environment, demonstrating a novel approach to uncovering bugs in low-level kernel components. This contributes to the broader theme of data infrastructure by showcasing advanced techniques for ensuring the robustness and security of the underlying operating system components that Cloudflare relies upon.

6/25/2019

This post announces partnerships with Chronicle Security, Datadog, Elastic, Looker, Splunk, and Sumo Logic to provide pre-built dashboards for analyzing Cloudflare logs and metrics. It highlights the integration with Logpush, which allows customers to push logs to S3-compatible storage, and the availability of these dashboards as Cloudflare Apps. The dashboards offer over 90 panels with insights into performance, security, and reliability.

5/18/2019

This post details Cloudflare's extensive use of eBPF (extended Berkeley Packet Filter) within its Linux-based edge network infrastructure. It highlights how eBPF is employed for critical functions such as volumetric DoS mitigation (XDP eBPF), layer 4 load balancing (XDP eBPF), and sophisticated socket filtering for UDP traffic. The post also touches upon custom kernel patches and iptables modules (TPROXY) used to overcome Linux networking limitations for advanced dispatching and handling of high volumes of traffic, showcasing a deep dive into kernel-level optimizations for network performance and security.

5/3/2019

This post details a critical bug discovered in the eBPF verifier related to 64-bit arithmetic operations (subtraction) when run by unprivileged users. It explains how a security mitigation for Spectre variant 1 attacks inadvertently broke this functionality, leading to incorrect calculations. The post demonstrates the bug, explains the underlying cause by examining kernel code and eBPF bytecode, and discusses workarounds like using 32-bit ALU operations or waiting for kernel patches. This contributes to the understanding of how Cloudflare leverages and debugs low-level kernel technologies for its infrastructure.

4/24/2019

This post introduces xdpcap, an open-source tool that acts as a replacement for tcpdump specifically for eXpress Data Path (XDP) programs. It enables packet capture and filtering at the kernel level, addressing the visibility gap created by XDP's early packet processing. The post details the technical implementation, including the use of cBPF to eBPF compilation via cbpfc and eBPF tail-calls for instrumentation, thereby enhancing the ability to analyze and debug network traffic at a very low level.

2/25/2019

Introduced Logpush, a new feature that allows enterprise customers to automatically push their Cloudflare request logs directly to cloud storage providers like Amazon S3 and Google Cloud Storage. This replaces the need for customers to write scripts using the Logpull API to download and then upload logs. Logpush maintains the same functionality as Logpull, including field selection, timestamp formatting, and random sampling, and offers a UI for setup.

1/4/2019

This post introduces the Linux AIO API as a potential alternative to epoll for network servers, focusing on its ability to batch syscalls like read and write. It explains how Linux AIO, traditionally for disk I/O, can be used with network sockets to reduce context switching overhead and improve performance, especially in high-traffic scenarios. The post also details how to bypass the `io_getevents` syscall by directly interacting with the kernel's ring buffer for completion events, further optimizing performance.

12/11/2018

This post analyzes traffic data from Cloudflare's network during Black Friday and Cyber Monday to understand their impact on e-commerce. It presents data on overall page views, e-commerce specific page views, checkout interactions, and device usage (mobile vs. desktop) across different regions (US, UK, Germany). The analysis highlights that while overall traffic doesn't show significant spikes, e-commerce traffic does, with mobile browsing dominating but desktop leading to more checkout interactions. This data provides insights into customer behavior during peak shopping periods.

11/29/2018

This post introduces an example Cloudflare Worker implementation that leverages sub requests to send traffic logs to an arbitrary location, specifically an Elastic stack. It details the Worker code for collecting log data (timestamp, URL, referrer, method, ray ID, IP, host, user agent, country code, colo, TLS version/cipher, status) and asynchronously posting it via a PUT request to an ELK endpoint. It also discusses how this enables real-time logging for all users, contrasting it with Enterprise Log Share (ELS) and suggesting improvements with Argo Tunnel and Access for securing the logging backend.

8/24/2018

Introduced ebpf_exporter, a tool that leverages eBPF to collect low-level system metrics, specifically focusing on disk I/O latency histograms. This addresses the limitations of traditional Prometheus exporters like node_exporter and cAdvisor by providing detailed event-level data, enabling more accurate performance analysis and alerting.

5/13/2018

This post details a deep dive into system-level performance issues encountered during an upgrade from Debian Jessie to Debian Stretch, specifically focusing on increased system CPU usage and RCU stalls on Kafka nodes. It highlights the use of `perf` for profiling and `dmesg` for identifying kernel-level issues like memory allocation stalls and dropped kernel messages, ultimately leading to the identification of a regression in the Linux kernel between Debian Jessie and Stretch as the root cause.

3/29/2018

This post introduces the use of eBPF (extended Berkeley Packet Filter) and socket options (`SO_ATTACH_BPF`) for advanced network introspection. It details how eBPF can be used to extract IP TTL values from TCP connections to determine hop distance, a novel approach to verifying traffic routing outside of the datacenter. The post also delves into the technical challenges and implementation details of writing eBPF assembly, including the use of eBPF maps for data sharing between kernel and userspace.

3/22/2018

Introduced the Cloudflare Mobile SDK for Android and iOS developers to visualize and understand their mobile app's network utilization. The SDK helps identify slowdowns caused by network errors, top N requests, slow requests, and third-party calls. It aims to correlate network experience data with existing engagement data tracked in tools like Mixpanel, Amplitude, and Heap. The SDK is free, has no cap on active users or metrics tracked, and does not require using Cloudflare's infrastructure. Privacy is a key consideration, with no collection of persistent identifiers and no selling of collected data.

3/6/2018

This post details the re-architecture of Cloudflare's HTTP traffic analytics pipeline, replacing a legacy PostgreSQL/Citus system with ClickHouse. It describes the challenges of the old pipeline (SPOFs, complexity, high maintenance cost) and the decision to adopt ClickHouse due to its performance, scalability, and fault tolerance. The post covers schema design for ClickHouse, including strategies for handling aggregated data due to high volume (6M requests/sec) and the trade-offs considered for storing raw vs. aggregated data. It also highlights Cloudflare's contributions to ClickHouse, such as the Kafka table engine and aggregate functions.

3/5/2018

This post details Cloudflare's experience with Kafka compression, specifically focusing on the improvements introduced in Kafka 0.11.0. It highlights the challenges faced with older versions and the benefits of the new batch compression strategy. The post includes performance testing of different compression codecs (gzip, lz4, snappy) on various data types (nginx errors, capnp-encoded requests) and analyzes the trade-offs between CPU, disk, and network usage. It also touches upon the integration of these improvements into their Go-based Kafka clients (sarama) and identifies potential bottlenecks in the compression process.

2/23/2018

This post introduces the application of Cloudflare Load Balancer to manage multi-cloud Kubernetes workloads, enabling intelligent traffic distribution, avoiding vendor lock-in, and improving cost-effectiveness. It highlights the ability to achieve a 'single pane of glass' for these distributed systems.

2/20/2018

This post introduces the capability of using Go as a scripting language in Linux by leveraging the `binfmt_misc` kernel module. It details the challenges with the traditional shebang approach for Go scripts and provides a practical solution using a custom interpreter (`gorun`) and kernel module registration to enable direct execution of `.go` files, improving prototyping and development efficiency.

1/18/2018

This post details the investigation and resolution of a mysterious processor bug affecting Intel Xeon E5-2650 v4 (Broadwell) processors. It highlights the process of identifying and debugging hardware-level issues that manifest as unexpected software crashes (SIGSEGV, SIGABRT, SIGILL) and the use of core dumps for post-mortem analysis. The discovery of the bug required meticulous examination of system logs and core dumps to rule out software bugs and eventually pinpoint the hardware as the source of the problem.

1/15/2018

This post delves into the low-level details of TCP SYN packet handling within the Linux kernel, specifically focusing on the SYN Queue and Accept Queue mechanisms. It explains how these queues function, their size limitations, the impact of application performance on connection handling, and the role of SYN cookies in mitigating SYN flood attacks. The post also highlights the use of tools like `ss` and custom SystemTap scripts for monitoring and debugging these network operations, contributing to Cloudflare's deep understanding and optimization of its network infrastructure.

12/22/2017

This post highlights several key developments in Cloudflare's data infrastructure and analytics capabilities. It details how Cloudflare analyzes 1 million DNS queries per second using a custom pipeline with ClickHouse and Grafana, and how they made their DNS stack 3x faster. It also touches on hardware performance comparisons (ARM vs. Intel), SystemTap for kernel analysis and performance optimization, and the analysis of DDoS attacks like WireX and reflection attacks. Additionally, it mentions the use of LavaRand for randomness generation and the impact of leap seconds on DNS.

11/13/2017

This post delves into performance optimization within Go programs, specifically addressing the impact of garbage collection (GC) on multi-core performance. It demonstrates how tuning the GOGC variable can drastically improve throughput for computationally intensive tasks by reducing the frequency and overhead of GC cycles, a critical consideration for high-performance systems.

11/10/2017

This post details a performance issue observed with Intel's AVX-512 instruction set, specifically its impact on the ChaCha20-Poly1305 cipher. It highlights how dynamic frequency scaling, triggered by AVX-512 usage, can significantly reduce overall CPU performance, even when AVX-512 is used for a small portion of the workload. The post contrasts the performance of OpenSSL and BoringSSL implementations and suggests disabling AVX-512 for general-purpose workloads to avoid unintended throttling, contributing to the understanding of low-level system performance and optimization strategies.

11/8/2017

This post introduces the evaluation of ARM-based processors (Qualcomm Centriq) as a potential alternative to Intel CPUs for Cloudflare's server infrastructure. It details performance benchmarks for cryptography and compression, compares hardware specifications, and assesses the ecosystem readiness of ARM for Cloudflare's software stack (NGINX, LuaJIT, Go). This marks an exploration into diversifying hardware choices for performance and cost optimization.

11/7/2017

This post delves into the optimization of network packet processing at the kernel level, specifically exploring the REUSEPORT socket option and its evolution with SO_INCOMING_CPU and SO_ATTACH_REUSEPORT_CBPF. It demonstrates how these features can improve packet locality by ensuring packets are processed on the same CPU that received them, thereby enhancing CPU cache locality and reducing cross-CPU interrupts. The post also introduces three advanced SystemTap scripts to measure and verify these improvements, though it notes that direct end-to-end performance gains for high-level TCP applications were not significantly measurable.

11/6/2017

This post details the technical implementation of LavaRand, a system that uses a wall of lava lamps as a source of true randomness for cryptographic operations. It explains the concepts of true vs. pseudorandomness, the importance of entropy pools in operating systems, and how LavaRand mixes physical entropy with local system entropy to create a more secure and unpredictable source for Cloudflare's production fleet, acting as an additional layer of defense against potential compromises of standard entropy sources or CSPRNGs.

11/6/2017

This post introduces LavaRand, a system that uses lava lamps as a secondary source of cryptographic randomness for Cloudflare's production servers. It explains the importance of unpredictable randomness in cryptography, how computers typically generate it, and how LavaRand leverages the unpredictable physical behavior of lava lamps, captured by a camera and fed into a CSPRNG, as a hedge against potential flaws in primary randomness sources. This adds a novel approach to data input for security-critical systems.

10/26/2017

This post details the integration of Cloudflare logs with Google Cloud Platform (GCP) to provide customers with an end-to-end solution for log analysis. It describes the use of GCP Storage for log storage, Cloud Functions for data import into BigQuery, and Data Studio for visualization. The solution allows customers to retrieve Cloudflare access logs via a REST API and process them for insights into traffic, security incidents, and infrastructure health.

10/23/2017

This post delves into the intricacies of load balancing in NGINX, specifically addressing the problem of uneven distribution of incoming connections across worker processes. It explains the different models for handling accept() calls, the Linux kernel's differing load balancing behaviors for blocking accept() vs. epoll-and-accept, and introduces SO_REUSEPORT as a solution for achieving more even load distribution. The post also highlights the critical trade-off between even load balancing and latency, demonstrating how SO_REUSEPORT can sometimes lead to degraded latency distributions under high load.

7/4/2017

This post introduces three small, open-sourced command-line tools: `mmhistogram` for generating ASCII histograms from data, `mmwatch` for observing the rate of change in command output, and `mmsum` for summing lists of floating-point numbers. These tools provide developers with convenient ways to quickly analyze and visualize data directly from the command line, contributing to the operational analytics capabilities.

7/3/2017

This post introduces PAL (Permissive Action Link), a tool developed by Cloudflare to securely distribute secrets (API tokens, private keys, passwords) to Dockerized production applications. It addresses the challenges of managing secrets in CI/CD workflows and containerized environments by providing a service identity bootstrapping mechanism. PAL allows encrypted secrets to be decrypted at runtime after a service's identity has been established, working with orchestration frameworks like Mesos and supporting encryption methods like PGP and Red October. The post highlights the importance of secure secret management for production systems and the decision to open-source PAL.

5/10/2017

This post details the implementation of a real-time analytics system for Cloudflare DNS logs, focusing on the challenges of handling high-volume, high-cardinality data. It describes the data pipeline from edge log forwarding via Cap'n Proto and Kafka to storage and analysis using ClickHouse. Key technical contributions include the development of a Go adapter for Kafka-to-ClickHouse ingestion, optimization of ClickHouse ingestion performance through batching, and careful selection of primary keys and materialized views for efficient querying of zone-level data and aggregations. The post also discusses the limitations of aggregated data and the necessity of unaggregated logs for root cause analysis.

5/3/2017

This post introduces how eero, an IoT device manufacturer, leverages Cloudflare's CDN and high availability to provide a resilient and cost-effective method for its devices to check internet connectivity. This acts as a 'cloud canary' and helps eero monitor its devices' connection to the cloud without privacy concerns or reliance on third-party sites. It highlights Cloudflare's role in supporting the specific needs of Internet of Things products and ensuring scale across millions of devices.

2/21/2017

This post details efforts to optimize LuaJIT performance by addressing 'Not Yet Implemented' (NYI) features, specifically focusing on the `next()` function used for table iteration. It introduces a new tool called 'loom' for better visualization and analysis of LuaJIT's JIT compilation process, and demonstrates how to improve the compilation of `next()` to avoid performance penalties associated with interpreter fallback.

2/14/2017

This post introduces a Grafana plugin for Cloudflare DNS analytics, enabling users to visualize DNS traffic patterns, response codes (including NXDOMAIN increases during DDoS attacks), and query breakdowns by data center and type. It also highlights the underlying DNS analytics API and provides installation instructions for the plugin and example curl commands for API usage. Virtual DNS customers can also leverage a custom dashboard for traffic distribution and RTT from origins.

1/1/2017

This post details a critical incident where a leap second caused a negative time value in Cloudflare's DNS resolution software (RRDNS), leading to a temporary outage for customers using CNAME records. It highlights a specific bug in the Go programming language's time handling and how Cloudflare's internal systems, particularly the weighted selection algorithm for upstream DNS resolvers, were affected. The post contributes to the understanding of Cloudflare's data infrastructure by showcasing the challenges of timekeeping in distributed systems and the engineering effort involved in identifying and fixing such subtle, yet impactful, issues.

12/28/2016

This post introduces the use of Guzzle and PHPUnit for automated REST API testing within a PHP environment. It details setting up the testing environment using Composer, configuring PHPUnit, and writing tests to simulate HTTP requests and validate responses. This contributes to the data infrastructure by providing a robust method for ensuring the quality and reliability of APIs, which are increasingly critical for modern internet services.

12/26/2016

This post details how to expose Go services directly to the internet by optimizing `crypto/tls` and `net/http` configurations. It provides specific guidance on setting secure TLS cipher suites, curve preferences, and implementing crucial timeouts (`ReadTimeout`, `WriteTimeout`, `IdleTimeout`) to ensure stability and prevent resource exhaustion. It also covers HTTP/2 considerations and best practices for `ServeMux` usage and logging, marking a shift from relying solely on reverse proxies to directly exposing Go applications with enhanced security and performance.

8/12/2016

This post details the debugging of a rare 522 error, tracing it back to a subtle issue where an application leaking sockets in the CLOSE_WAIT state on the loopback interface caused connection timeouts for new connections. It highlights how this application-level problem, not a network issue, can lead to symptoms that appear to violate TCP specifications, impacting Cloudflare's edge connectivity.

8/3/2016

This post introduces Cloudflare's use of JSON Schema and JSON Hyper-Schema for defining their RESTful API. It details the development of a toolchain, including `json-schema-loader`, `json-schema-example-loader`, and `doca`, to automatically generate HTML documentation from these schemas. This demonstrates an evolution in how Cloudflare manages and exposes its data and API, contributing to internal tooling and developer experience.

7/12/2016

This post details the scale of edge log processing at Cloudflare, handling millions of HTTP and DNS logs per second, resulting in hundreds of terabytes of raw data daily. It highlights the infrastructure used, including Kafka clusters and CitusDB, and discusses what has worked well (log forwarder, Kafka, Log Share persistence/retrieval, CitusDB, platform/SRE support) and what remains to be done (improving service reliability for customers, new analytics systems, new data pipelines, better support for complex analysis).

7/7/2016

This post delves into Cloudflare's strategic decision to utilize the Linux kernel's TCP stack, explaining the benefits of hardware independence, time-sharing, and running multiple network applications. It highlights the trade-offs of userspace networking and kernel bypass, particularly the loss of concurrent application support. The post details Cloudflare's use of 'partial kernel bypass' for DDoS mitigation, offloading iptables to a userspace process to avoid IRQ storms. It also emphasizes the value of the Linux TCP stack's features, debugging tools, and ecosystem over custom solutions.

1/19/2016

This post introduces a technique for capturing Go test coverage data from end-to-end tests, which is not natively supported by the Go toolchain. It details a workaround involving a dummy test that executes the main function and compiles a binary with coverage enabled. The post also explains how to merge this coverage data with unit test coverage using `gocovmerge` for a more complete picture of code execution, specifically mentioning its application in the RRDNS project.

12/17/2015

This post details the integration of Cloudflare's work on netmap into the mainline netmap repository. The key contribution is the development of a more configurable netmap mode that allows for partial kernel bypass, enabling specific network queues to be detached from the host stack while others remain attached. This allows for more flexible packet processing and the ability to capture transmitted packets by moving them to the RX host ring. New flags (`NR_TX_RINGS_ONLY`, `NR_RX_RINGS_ONLY`) were introduced for finer control over ring pair usage.

11/19/2015

This post details the investigation and resolution of rare, significant latency spikes experienced by a customer. It showcases the use of advanced debugging tools like System Tap and flame graphs to pinpoint the root cause within the Linux kernel's `tcp_collapse` function, which was exacerbated by large TCP receive buffer sizes. The post demonstrates how tuning the `net.ipv4.tcp_rmem` sysctl parameter significantly improved network performance and stability, highlighting Cloudflare's commitment to deep system-level performance optimization.

10/9/2015

This post details Cloudflare's contribution to the Netmap project, introducing a 'single RX queue mode'. This feature allows user-space applications to bypass the Linux kernel for specific network flows, enabling high-speed packet processing and filtering of large packet floods. This is crucial for their network infrastructure, especially when dealing with attacks, and addresses limitations with certain NICs by providing a more flexible kernel bypass solution.

9/7/2015

This post introduces the concept of 'kernel bypass' as a necessary technique to overcome the performance limitations of the vanilla Linux kernel networking stack for high-throughput packet processing. It details experiments demonstrating the kernel's limitations with multi-queue packet handling and then explores various kernel bypass techniques including PACKET_MMAP, PF_RING, Snabbswitch, DPDK, Netmap, Solarflare's EF_VI, bifurcated drivers, and virtualization approaches, highlighting their pros and cons in the context of Cloudflare's needs.

8/3/2015

This post introduces a technique for improving the debuggability of Go stack traces by converting IPv4 addresses to their uint32 representation and passing this as a blank identifier parameter to goroutines. This allows engineers to quickly identify which network interface a specific goroutine is listening on by examining the stack trace, significantly aiding in troubleshooting issues within their large-scale Go applications like RRDNS.

7/23/2015

This post introduces a new 'Analytics' section within the Partner Portal, providing granular data to hosting provider partners. This includes detailed breakdowns of bandwidth savings (cached vs. uncached), threat mitigation (total threats, top countries, top threat types), and request/visitor metrics (total requests, cached requests, total unique visitors). This is enabled by an upgrade to the customer analytics and request logging backend, allowing for distributed queries and dynamic data delivery to over 5,000 partners.

7/3/2015

This post introduces a novel, visual alerting system called the 'Blue Light Special' to monitor and ensure the rapid global propagation of configuration changes (like DNS records and WAF rules) across Cloudflare's network. It highlights the challenge of near-instantaneous updates and the use of a Raspberry Pi connected to a visible police light to provide a visceral, real-time indicator of propagation times, ensuring developers and operations teams are immediately aware of any delays.

7/1/2015

This post introduces a more efficient method for embedding version information into Go binaries using the `-X` linker option, replacing the previous Makefile-based generation of `version.go`. This demonstrates an improvement in build tooling and a smarter approach to managing application metadata within the Go ecosystem.

6/18/2015

This post introduces the `godebug` tool, a cross-platform debugger for Go programs. It highlights how Cloudflare uses `godebug` to debug its internal Go DNS server, RRDNS, demonstrating its effectiveness in improving development workflows and the ability to debug complex Go applications without requiring additional tools or permissions on the server. The post also touches upon the broader concept of source rewriting in Go tooling.

5/11/2015

This post contributes to the understanding of Cloudflare's deep engagement with low-level network protocols (TCP/IP stack, IPv4/IPv6, UDP, ICMP) by sharing interview questions. It reveals that Cloudflare engineers are expected to have a comprehensive understanding of these protocols, which is crucial for developing and maintaining their automatic attack mitigation systems. The post touches upon various aspects of network communication, including packet structure, flags, fragmentation, congestion control, and Linux-specific network configurations.

5/7/2015

This post details Cloudflare's significant investment in optimizing Go's cryptography performance. It highlights the creation of assembly implementations for Elliptic Curves and AES-GCM to match or exceed OpenSSL's speed, addressing a critical need for high-volume encryption at Cloudflare's scale. The post also emphasizes the contribution of these performance improvements back to the Go community through a special fork, demonstrating a commitment to open source and shared advancement in cryptographic performance.

4/9/2015

This post details the significant evolution of Cloudflare's data infrastructure for analytics, specifically addressing the scaling challenges of their log processing pipeline. It introduces the adoption of CitusDB to scale PostgreSQL beyond a single instance, enabling real-time analytics with sub-second query responses across billions of rows. The post also elaborates on the pipeline architecture, including the use of Nginx with Lua, Cap'n Proto, Go programs, and Kafka, highlighting the rationale behind choosing each technology for performance, persistence, and future extensibility.

3/25/2015

This post highlights a common pitfall in Go programming when using closures with goroutines, specifically demonstrating how shared loop variables can lead to unexpected behavior. It provides a clear explanation and a practical solution by passing the loop variable as a function parameter, contributing to the understanding of robust concurrent programming practices within Cloudflare's data infrastructure.

9/15/2014

This post details Cloudflare's use of Go and its approach to managing goroutine stacks. It explains the evolution from segmented stacks to stack copying, highlighting the performance implications and the ongoing efforts to rewrite the Go runtime in Go to enable features like concurrent garbage collection and improve stack copying capabilities. This contributes to the understanding of how Cloudflare optimizes its internal systems for performance and resource efficiency.

8/27/2014

This post introduces the use of Go interfaces to simplify unit testing for components like image compression workers. By defining a `Job` interface, the `Worker` package can accept any type that implements `Do()` and `Priority()`, allowing for the creation of `DummyJob` implementations for testing purposes. This significantly improves test isolation and reduces the complexity of setting up test environments for components that interact with complex underlying systems.

3/5/2014

This post delves into the specifics of how Go programs on Linux obtain the current time, highlighting the evolution from older system calls like `time` and `gettimeofday` to the more precise `clock_gettime`. It details Cloudflare's adoption of the Linux vDSO (Virtual Dynamically linked Shared Objects) mechanism for low-overhead time retrieval, significantly improving performance for `time.Now()` and `runtime.nanotime`. The post also includes benchmarks demonstrating the performance gains and discusses the importance of accurate timekeeping for various system functions and potential pitfalls like Y2K and leap second bugs.

8/24/2013

This post delves into memory management challenges in Go, a language heavily used at Cloudflare for network services. It details techniques for recycling memory buffers using channels and a dedicated recycler goroutine to reduce garbage collection overhead and improve memory efficiency, demonstrating a practical application of low-level optimization within Cloudflare's data infrastructure.

7/22/2013

This post details the evolution of Cloudflare's server hardware (G1-G4) and its components (CPU, storage, network) from 2010 to 2013. It highlights the specific design choices and optimizations made to handle the massive scale of traffic, including the adoption of SSDs, increased RAM for caching, low-power Intel Xeon CPUs, and high-performance Solarflare network cards with kernel bypass technology. This hardware advancement is crucial for enabling the data processing and analytics capabilities discussed in other posts.

7/8/2013

This post details the technical challenge of propagating configuration rules across Cloudflare's global network of Points of Presence (PoPs) in near real-time. It introduces the use of PostgreSQL for storing these rules and Kyoto Tycoon (a distributed key-value store) for replication. The core contribution is the implementation of a writable Foreign Data Wrapper (FDW) for PostgreSQL 9.3, allowing direct integration and transactional operations with Kyoto Tycoon, thereby simplifying and accelerating the rule synchronization process.

4/23/2013

This post details Cloudflare's initial logging policies and infrastructure. It explains that logs are generated for security threat identification and performance bottleneck analysis. For most customers, logs are discarded within 4 hours due to the massive scale (10 Petabytes/year and doubling every 4 months). Enterprise customers can opt to store logs for 3 days for export. Aggregate data is used for analytics (Cloudflare Analytics page) and security rule updates, with a stream processing engine correlating data at edge data centers. Error logs are kept for 1 week.

1/11/2013

This post details a significant system-wide upgrade focused on migrating Cloudflare's customer provisioning and accounting systems. The upgrade involved moving over half a million websites to a new, more modern system, rebalancing them across IP addresses, and seamlessly reissuing SSL certificates. This migration significantly enhanced the network's flexibility and robustness, enabling better isolation of sites under attack and facilitating compliance with regional content restrictions. It represents a major step in the operational maturity of Cloudflare's data infrastructure, allowing for more dynamic resource allocation and improved management of network resources.

12/8/2012

This post introduces the use of Lua scripting within Nginx to augment its capabilities. It details how Lua can be embedded into Nginx to add custom functionalities, such as real-time log aggregation and processing, without resorting to custom C modules. This demonstrates an evolution in how Cloudflare processes and analyzes operational data at the edge, complementing its existing data infrastructure and analytics platforms by enabling more flexible and performant custom logic execution.

5/28/2012

Refined the page view counting algorithm to more accurately distinguish between page views and other content types (images, non-HTML) and to exclude certain redirects that caused double counting. This change results in a more precise reporting of page views, bringing Cloudflare's numbers closer to beacon-based tracking systems while still capturing a more comprehensive view of total server load.

8/29/2011

This post details a change to Cloudflare's logging system to exclude AJAX (XHR) calls from page view counts. Previously, these calls were treated as page views, leading to inflated numbers for sites with heavy AJAX usage. The update modifies how logs are counted at the network edge, resulting in a system-wide drop of approximately 17% in reported page views, with a more substantial decrease for AJAX-heavy sites to provide a more accurate representation.

7/8/2011

This post clarifies Cloudflare's approach to analytics, differentiating its raw log-based hit and bandwidth tracking from beacon-based page view tracking (like Google Analytics). It explains the challenges of defining 'page views' in the context of modern AJAX-driven websites and highlights that Cloudflare's metrics provide a more accurate picture of server load and resource consumption, while acknowledging that beacon-based services are better suited for ad impression tracking.

5/18/2011

This post marks a significant milestone in Cloudflare's operational scale, achieving 2,000 page views per second. It highlights the evolution of their data infrastructure from manual log monitoring to a sophisticated, real-time global traffic display, demonstrating an increased capacity for handling and visualizing massive amounts of network data. This reinforces the ongoing development of their data infrastructure and analytics capabilities to manage and understand network performance at scale.

4/21/2011

This post introduces the Cloudflare API's `zone_ips` function, allowing customers to programmatically retrieve visitor IP addresses from the past 48 hours. It details how to use the API with parameters for API key, zone ID, hours, and optional class (regular, crawler, threat) and geo-location data. The post also notes differences in data availability for Free vs. Pro accounts and suggests using the data for threat detection and blocking IPs via the Threat Control panel.

12/17/2010

This post details an upgrade to Cloudflare's core logging infrastructure to improve the speed of displaying statistics. An I/O bottleneck was identified and addressed in the short term by integrating Fusion-IO cards into the stats database, moving fast-access data and rebuilding indexes. This upgrade resulted in significantly faster stats page loading and query performance. The company is evaluating the long-term viability of this hardware for its core storage architecture.