TechPrufer
Get started
Cloudflare
Cloudflare logoCloudflare · Feature trail

Data Infrastructure & Analytics

301 posts · Dec 2010 – Oct 2026

What this trail covers

Cloudflare Basin is a serverless data analytics platform built on Apache Iceberg and R2 Object Storage. It includes Basin Pipelines for event ingestion and transformation, Basin Catalog for managing Iceberg metadata and performing automatic maintenance, and Basin SQL for serverless querying of Iceberg tables. The platform emphasizes speed, openness, and cost efficiency, with usage-based pricing and zero egress fees. Future plans include custom partitioning, schema migrations, support for Iceberg V3, and enhanced jurisdiction controls.

All Cloudflare trails
  1. Introducing Cloudflare Basin: an open, serverless data platform, now generally availableThis post announces the general availability of Cloudflare Basin, formerly the Cloudflare Data Platform. It details the three core components: Basin Pipelines (formerly Cloudflare Pipelines) for event ingestion and transformation using SQL into Apache Iceberg tables or R2 files; Basin Catalog (formerly R2 Data Catalog) for managing Iceberg metadata and performing automatic maintenance like compaction and snapshot expiration; and Basin SQL (formerly R2 SQL) as a serverless query engine for Iceberg tables. Key improvements since beta include increased ingestion scalability for Pipelines (up to 3GB/s), Logpush integration, schema-aware Worker bindings, data quality error visibility, Terraform support for IaC, and new features for Catalog like per-table compaction policies and unreferenced data-file cleanup. Future plans for Pipelines include custom partitioning, schema migrations, and stateful processing. Future plans for Catalog include improved compaction sorting/clustering, granular auth controls, and jurisdiction support.
  2. Introducing the Billable Usage API: programmatic cost visibility for CloudflareIntroduced the Billable Usage API, a new endpoint that provides programmatic access to Cloudflare account usage and cost data. The API returns data broken down by product and service period, aligning with the FinOps Open Cost and Usage Specification (FOCUS). This enables automated cost tracking, reporting, and integration with FinOps toolchains.
  3. Announcing the Monetization Gateway: charge for any resource behind Cloudflare via x402This post introduces the Cloudflare Monetization Gateway, a new capability that allows customers to charge for any resource protected by Cloudflare using usage-based pricing and micropayments via the x402 protocol and stablecoins. This extends Cloudflare's data infrastructure and analytics by enabling granular metering and payment enforcement at the edge, addressing the evolving business model of the web driven by AI agents and the need for efficient, low-cost transaction rails for sub-cent payments.
  4. How we built Cloudflare's data platform and an AI agent on top of itThis post details the creation of Cloudflare's unified data analytics platform, Town Lake, and an AI data agent, Skipper. Town Lake unifies disparate data sources (Postgres, ClickHouse, R2, Kafka, BigQuery) into a single SQL interface using Apache Trino as the query engine, R2 Data Catalog (Apache Iceberg) for storage, and DataHub for metadata. It implements a 'default-closed' governance model with automated PII scanning (Skimmer) and access control (Lifeguard). Transformer, built on Workflows, handles ELT. Skipper, built on Workers AI and Durable Objects, provides a natural language interface to Town Lake, capable of generating SQL, performing closed-loop reasoning, and creating dashboards. The platform is built entirely on Cloudflare's own products (R2, Workers, Access, Workflows).
  5. Iran's Internet is partially restored, Cloudflare Radar data showsThis post details the use of Cloudflare Radar data to analyze and confirm the partial restoration of Internet access in Iran following a prolonged shutdown. It examines traffic volume, regional distribution, network provider impact, and DNS query increases, providing specific metrics and observations. It also highlights the continued impact on IPv6 connectivity compared to IPv4, suggesting the shutdown was achieved through application filtering or whitelisting rather than routing table manipulation.
  7. Our billing pipeline was suddenly slow. The culprit was a hidden bottleneck in ClickHouseThis post details the debugging and resolution of performance degradation in the Ready-Analytics ClickHouse cluster due to increased data part counts following a migration to a new partitioning scheme. Three patches were developed and contributed upstream to address lock contention during query planning (using shared locks instead of exclusive locks), vector copying (deferring the copy and creating a shared copy), and to improve part filtering efficiency (binary search for parts).
  8. When "idle" isn't idle: how a Linux kernel optimization became a QUIC bugThis post details a critical bug discovered in the CUBIC congestion control algorithm, specifically how a Linux kernel optimization, when ported to Cloudflare's quiche QUIC implementation, caused the congestion window to get permanently pinned at its minimum after a congestion collapse event. The post explains the root cause, tracing it back to an optimization for handling application idle periods and how its implementation in user-space QUIC differed from the kernel's TCP, leading to unexpected oscillations and failure to recover bandwidth. It highlights the importance of rigorous testing in edge cases and the successful resolution of the bug.
  9. Artifacts: versioned storage that speaks GitThis post introduces Artifacts, a new primitive: a distributed, versioned filesystem built for agents that speaks Git. It allows programmatic creation of Git repositories for agents, sandboxes, and other compute paradigms, with REST API and native Workers API access. It also introduces ArtifactFS, a filesystem driver for mounting large Git repos quickly by hydrating file contents on the fly, ideal for agents and sandboxes with critical startup times. Artifacts is built on Durable Objects and a custom Git server implementation in Zig compiled to Wasm.
  10. Evolving Cloudflare’s Threat Intelligence Platform: actionable, scalable, and ETL-lessThis post introduces a significant evolution of Cloudflare's Threat Intelligence Platform (TIP), moving away from complex ETL pipelines to a sharded, SQLite-backed architecture running GraphQL directly on the edge. This enables real-time visualization and automation of threat response with sub-second query latency over millions of events. The platform unifies global telemetry with analyst investigations to provide a single source of truth for proactive threat blocking. It complements SIEMs by providing specialized schema and long-term retention for deep adversary tracking, enriching raw logs with historical actor patterns and enabling faster, more accurate decision-making. The architecture eliminates bottlenecks through parallel execution at the edge using SQLite-backed Durable Objects and Cloudflare Workers, with dynamic visualizations like Sankey Diagrams and attribute mapping for correlating threat events. Saved configurations and real-time notifications facilitate proactive threat hunting, and automated rules and STIX2 exports enable seamless integration with SIEM/SOAR platforms.
  11. ASPA: making Internet routing more secureIntroduced a new ASPA deployment monitoring feature within Cloudflare Radar to track the adoption and trends of the ASPA standard across Regional Internet Registries (RIRs) and Autonomous Systems (AS). Detailed the technical implementation and validation process of ASPA, including its role in securing Internet routing by verifying the AS_PATH and preventing route leaks. Provided a practical example of creating ASPA objects within RIPE and ARIN dashboards.
  12. Improve global upload performance with R2 Local UploadsIntroduced Local Uploads for R2, enabling object data to be written to a client-proximate storage location before asynchronous replication to the bucket's primary region. This reduces upload Time to Last Byte (TTLB) by up to 75% for cross-region uploads. The implementation uses Cloudflare Queues for managing asynchronous replication tasks and a pull model for consuming these tasks via Gateway Workers. Metadata publishing is atomic, involving storing metadata, creating pending replica keys, and scheduling replication tasks.
  13. Announcing support for GROUP BY, SUM, and other aggregation queries in R2 SQLThis post introduces support for aggregation queries (GROUP BY, SUM, COUNT, AVG, etc.) in R2 SQL. It details the implementation of 'scatter-gather aggregations' for queries without HAVING/ORDER BY clauses, which compute pre-aggregates on worker nodes and merge them on the coordinator. For queries with HAVING/ORDER BY, it introduces 'shuffling aggregations' that use deterministic hash partitioning to colocate data for specific groups before final aggregation, followed by a synchronization barrier and local finalization on workers, and a k-way merge on the coordinator. This significantly enhances R2 SQL's analytical capabilities for large datasets.
  14. The 2025 Cloudflare Radar Year in Review- the rise of AI, post-quantum, and record-breaking DDoS attacksThis post introduces the 2025 Cloudflare Radar Year in Review, detailing observed Internet trends. Key contributions include data on global traffic growth, AI crawling patterns, post-quantum encryption adoption, connectivity metrics, and security threats. It also highlights new data sets for speed tests and DDoS attack sizes, and specific AI model usage on Workers AI.
  15. Fresh insights from old data: corroborating reports of Turkmenistan IP unblocking and firewall testingThis post details the use of Cloudflare Radar's historical data on TCP connection resets and timeouts to corroborate reports of IP unblocking and potential firewall testing in Turkmenistan. It analyzes trends in request volume and TCP anomaly patterns across different autonomous systems within Turkmenistan to provide insights into network behavior changes.
  16. Measuring characteristics of TCP connections at Internet scaleThis post shares aggregate insights about TCP connection characteristics observed through Cloudflare's global CDN. It details the methodology for collecting and analyzing socket-level metadata from the Linux kernel's TCP_INFO struct, focusing on connections that close gracefully with a FIN packet and have at least one successful HTTP request. The analysis presents empirical data on packet counts (response packets, and packet ratios between client and server) and bytes sent, highlighting the heavy-tailed nature of internet traffic and differences across HTTP protocol versions. This data is intended to inform simulations and provide a deeper understanding of real-world network behavior.
  17. The tricky science of Internet measurementThis post introduces the scientific practice of Internet measurement as a critical component of Cloudflare's data infrastructure and analytics. It highlights the inherent opacity of the Internet and the unique challenges it presents for measurement. The post details key measurement concepts like active vs. passive and direct vs. indirect measurement, and outlines the measurement lifecycle (data curation, modeling, validation). It uses an internal example of analyzing HTTP traffic spikes in Ukraine to illustrate the importance of context and avoiding erroneous conclusions, emphasizing the need for rigor, repeatability, and reproduction in measurement.
  18. From .com to .anything- introducing Top-Level Domain (TLD) insights on Cloudflare RadarIntroduces a new TLD page on Cloudflare Radar, providing insights into TLD popularity, activity, and security. This includes a ranking of top-level domains based on DNS Magnitude, a metric that estimates a domain's overall visibility on the Internet by incorporating the number of unique clients querying domains within a TLD. The post also details the methodology behind DNS Magnitude calculation and the types of TLDs (gTLD, grTLD, ccTLD, iTLD, sTLD). Additionally, it describes dedicated per-TLD pages offering general information (type, manager, DNS magnitude, DNSSEC/RDAP support) and WHOIS data, with direct links to Cloudflare Registrar for supported TLDs.
  19. Internet measurement, resilience, and transparency: blog takeover from Cloudflare Research and friendsThis post details foundational concepts and tradeoffs in Internet measurement, discusses the evolution of Cloudflare Radar and the Internet speed test, and introduces a framework for Internet resilience. It also covers advancements in post-quantum cryptography, including Merkle Tree Certificates, and explores optimizations in Cloudflare's Linux networking stack for addressing space and performance. Additionally, it details methods for CGNAT detection, DDoS defense, and cache efficiency. The post also introduces cryptographic protocols for an agentic web, building on Web Bot Auth, to enable secure identification of automated clients and protect websites from various threats, with a focus on post-quantum readiness.
  20. Making the Internet observable- the evolution of Cloudflare RadarThis post details the evolution of Cloudflare Radar over five years, highlighting its expansion into Internet security transparency with Certificate Transparency (CT) data, TCP resets and timeouts for detecting connection tampering, and post-quantum encryption adoption tracking. It also introduces AI Insights, providing visibility into AI crawler activity, crawl-to-refer ratios, and robots.txt usage by AI bots. Furthermore, Radar has enhanced routing visibility with data on route leaks and origin hijacks, contributing to Internet resilience.
  21. Monitoring AS-SETs and why they matterIntroduced AS-SET information to Cloudflare Radar's routing section, providing a public listing for network operators. This feature allows monitoring of AS-SET memberships, helps in building BGP route filters using tools like bgpq4, and demonstrates how accurate AS-SETs prevent route leaks. The feature also highlights AS-SET misuse and provides enhanced AS-SET data with inferred ASN, IRR sources, member counts, AS Cone, and upstream information. A tree-view visualization for AS-SET inclusion is also available.
  22. Introducing new regional Internet traffic and Certificate Transparency insights on Cloudflare RadarIntroduced regional traffic insights to Cloudflare Radar, enabling localized analysis of Internet traffic trends. This includes breaking down traffic by bytes, requests, desktop/mobile device shares, and bot/human traffic shares. Also added detailed Certificate Transparency (CT) data to Radar. Enhanced the Data Explorer to allow for combined regional and ASN analysis, and added new summary and time series views for regional traffic characteristics on country-level pages.
  23. Announcing the Cloudflare Data Platform: ingest, store, and query your data directly on CloudflareThis post announces the launch of the Cloudflare Data Platform, comprising Cloudflare Pipelines, R2 Data Catalog with compaction, and R2 SQL. Cloudflare Pipelines, built on Arroyo, ingests and transforms events using SQL, sinking them to R2 or R2 Data Catalog with exactly-once guarantees. R2 Data Catalog now includes automatic compaction for small files to improve query performance. R2 SQL is an in-house distributed SQL engine for querying data in R2 at petabyte scale. The platform emphasizes open standards, zero egress fees, and usage-based pricing.
  24. R2 SQL: a deep dive into our new distributed query engineThis post details the architecture and techniques behind R2 SQL, a serverless query engine for R2 object storage. It introduces the Query Planner, which uses Iceberg metadata (partition-level and column-level stats) to prune irrelevant data, and the Query Execution system, which distributes work across Cloudflare's network. Key innovations include a streaming planning pipeline that emits work units as they become available and a 'finish early' strategy enabled by ordering work units based on the query's ORDER BY clause and metadata statistics, allowing queries to complete after processing only a fraction of the data.
  25. Reducing double spend latency from 40 ms to < 1 ms on privacy proxyThis post details the optimization of a critical low-latency operation within Cloudflare's privacy proxy product. By analyzing tracing and metrics data, engineers identified a 40ms latency bottleneck in double-spend checks for Privacy Pass tokens. Through detailed investigation and hypothesis testing, they pinpointed the root cause to be the interaction of Nagle's algorithm and delayed ACKs in TCP, exacerbated by the Linux kernel's HZ setting. The fix involved implementing a `BufWriter` to buffer writes and send commands as single messages, reducing latency to under 1ms. This demonstrates a deep dive into network protocol optimization for performance-critical services.
  26. Building Jetflow: a framework for flexible, performant data pipelines at CloudflareThis post introduces Jetflow, a new internal framework built by the Cloudflare Business Intelligence team to address limitations in their existing ELT solution for petabyte-scale data ingestion. Jetflow achieves over 100x efficiency improvement in GB-s and >10x performance improvement in row ingestion rates. Key technical contributions include a modular pipeline design with Consumers, Transformers, and Loaders, the use of Apache Arrow for in-memory columnar data representation to minimize serialization overhead and GC pressure, and a data division strategy (RunInstance, Partition, Batch) for idempotency and efficient retries. It also simplifies schema management by inferring schemas from source systems and supports a wide range of data sources and targets.
  27. Explore your Cloudflare data with Python notebooks, powered by marimoThis post announces a partnership with marimo to integrate their reactive Python notebook environment with Cloudflare's data services. Key contributions include: 1. Cloudflare authentication built into marimo notebooks, allowing users to sign in with their Cloudflare account and use Cloudflare APIs without API tokens. 2. Open-source notebook examples for exploring Cloudflare data (R2, Workers AI, D1). 3. The ability to run marimo notebooks on Cloudflare Containers for scalable data workflows. The integration leverages WebAssembly for in-browser execution and PKCE for secure OAuth authentication.
  28. How TimescaleDB helped us scale analytics and reportingThis post details the decision to use TimescaleDB, a PostgreSQL extension, over ClickHouse for the analytics plane of Cloudflare's Digital Experience Monitoring (DEX) product. The author emphasizes simplicity and avoiding unnecessary complexity, advocating for starting with fewer components. The initial DEX architecture used PostgreSQL for configuration and a simple table for logs. When scaling the analytics plane, ClickHouse was considered but deemed overly complex for the MVP's needs (millions of small inserts every 2 minutes). TimescaleDB was chosen as a simpler, more familiar solution that could handle the time-series data effectively within the existing PostgreSQL ecosystem. The post also discusses the importance of multicolumn index order in PostgreSQL for time-series workloads and provides an example of a typical query and the corresponding index design.
  29. Russian Internet users are unable to access the open InternetThis post details the impact of ISP-level throttling in Russia on Cloudflare's network and users. It analyzes traffic trends using Cloudflare Radar, showing a significant decrease in traffic served to Russian users. It also presents Network Error Logging (NEL) data indicating increased TCP resets and QUIC protocol errors. Furthermore, internal data analysis using packet loss metrics and TCP resets and timeouts (specifically the 'Post PSH' stage) corroborates the throttling mechanism, which limits data transfer to 16 KB.
  30. Cloudflare service outage June 12, 2025This post details a significant service outage on June 12, 2025, caused by a failure in the underlying storage infrastructure for Workers KV, a critical dependency for many Cloudflare products. The outage impacted a wide range of services including Workers KV, WARP, Access, Gateway, Images, Stream, Workers AI, Turnstile, AutoRAG, Zaraz, and parts of the Dashboard. The incident highlighted the fragility of relying on a central data store for Workers KV and the cascading effects of such failures across the platform, emphasizing the urgent need for the ongoing transition to more resilient infrastructure.
  31. Bringing connections into view: real-time BGP route visibility on Cloudflare RadarIntroduced a new real-time BGP route lookup service accessible through the Cloudflare Radar web interface and API. This service provides visualizations of real-time routes on IP prefix pages, including Sankey diagrams illustrating routes to Tier 1 networks and options to show full paths. It also displays prefix origin tables with visibility percentages and RPKI validation outcomes. The Cloudflare Radar API now offers programmatic access to this real-time BGP route data, enabling automated tools and systems.
  32. Performance measurements… and the people who love themThis post introduces new tools and techniques for analyzing performance data, revealing surprising insights into system behavior and the impact of statistical aggregation on understanding true latency. It also clarifies definitions for performance metrics like TTFB and discusses the limitations of aggregated data in representing actual latency.
  33. How the April 28, 2025, power outage in Portugal and Spain impacted Internet traffic and connectivityThis post details the impact of a major power outage in Portugal and Spain on internet traffic, network quality, and routing, providing granular data analysis from Cloudflare's global network perspective. It includes country-level, network-level, and regional traffic drops, as well as changes in network quality (download speeds, latency) and routing (announced IP address space).
  34. Making Super Slurper 5x faster with Workers, Durable Objects, and QueuesRearchitected Super Slurper using Cloudflare Workers, Durable Objects, and Queues to improve data transfer speeds by up to 5x. Addressed performance bottlenecks in the original Kubernetes-based architecture by leveraging edge compute for object copying, distributed queues for concurrency, and Durable Objects (with SQLite) for state management and sharding. Implemented robust failure handling with retries and a Dead Letter Queue. Utilized Hyperdrive for accessing legacy PostgreSQL data. Introduced sequence IDs for listing operations and object key locking in the database to handle duplicate messages and ensure single transfers.
  35. Just landed: streaming ingestion on Cloudflare with Arroyo and PipelinesThis post announces the open beta of Cloudflare Pipelines, a streaming ingestion product that allows high-volume, structured, real-time data ingestion directly into R2 object storage. It also announces the acquisition of Arroyo, a cloud-native stream processing engine, to enable real-time data transformation, aggregation, and SQL-based querying on ingested data. Pipelines is built using Durable Objects and embedded SQLite for immediate persistence and replication via the Storage Relay Service (SRS), enabling horizontal scalability and high availability. Data is batched, compressed, and written to R2 in Hive-partitioned, newline-delimited JSON (NDJSON) files with ULID filenames. The post details the architecture of Pipelines, including the separation of control plane and data path, and the flow of records from ingestion to R2.
  36. R2 Data Catalog: Managed Apache Iceberg tables with zero egress feesIntroduced R2 Data Catalog in open beta, a managed Apache Iceberg catalog built directly into Cloudflare R2 buckets. This provides database-like capabilities (ACID transactions, schema evolution, efficient querying) for data stored in object storage without requiring external data catalog infrastructure. It exposes a standard Iceberg REST catalog interface for integration with engines like PyIceberg, Snowflake, and Spark. Pricing for the catalog operations and data compaction will be introduced in the future, while R2 storage and operations costs remain unchanged and egress fees are $0.
  37. A steam locomotive from 1993 broke my yarn testThis post details a deep dive into a persistent testing failure within Cloudflare's internal Backstage instance on Linux, characterized by a consistent 27-second timeout leading to a cryptic '[Error]'. The troubleshooting process involved extensive use of shell history, Jest flags, and ultimately `strace`, revealing an underlying system-level issue related to `epoll_wait` and `SIGCHLD` signals, potentially linked to older system components or configurations. This investigation highlights the challenges of maintaining robust development environments and the importance of low-level system analysis for debugging.
  38. Cloudflare incident on March 21, 2025This post details a significant incident impacting R2 object storage and several dependent services (Cache Reserve, Images, Log Delivery, Stream, Vectorize) due to a credential rotation error. It highlights the critical need for robust visibility into authentication mechanisms and proper environment management in production systems, directly impacting the reliability and availability of data infrastructure and analytics services.
  39. Extending Cloudflare Radar’s security insights with new DDoS, leaked credentials, and bots datasetsThis post introduces several new datasets and visualizations to Cloudflare Radar's security insights: new DDoS-focused graphs with geographic and AS-level distribution maps, new insights into leaked credential trends with a dedicated graph showing clean vs. compromised requests, and a new dedicated Bots page for monitoring bot traffic vs. human traffic. It also details the refactoring of the Security & Attacks page into separate Application Layer and Network Layer sections.
  40. Some TXT about, and A PTR to, new DNS insights on Cloudflare RadarIntroduced a dedicated DNS page on Cloudflare Radar, leveraging data from the 1.1.1.1 resolver and the AS112 Project. This page provides detailed insights into DNS traffic trends, including global, location, and ASN traffic, protocol usage (UDP, DoT, DoH, TCP), query and response characteristics (record types like A, AAAA, HTTPS), and DNSSEC usage. The post details the analysis of this data, including its use for identifying ccTLD popularity and corroborating reported Internet outages or blocking of 1.1.1.1.
  41. Searching for the cause of hung tasks in the Linux kernelThis post delves into the Linux kernel's 'hung task' warnings, explaining their meaning, how the kernel identifies them, and how Cloudflare configures monitoring for these events. It provides detailed examples of debugging hung tasks related to XFS filesystem issues and coredump generation, highlighting how these warnings can surface underlying system performance problems or unexpected behaviors, thereby contributing to the broader understanding and optimization of Cloudflare's data infrastructure.
  42. Automatic Audit Logs: new updates deliver increased transparency and accountabilityIntroduced Automatic Audit Logs, a new system that standardizes and automates audit log generation across Cloudflare products. This system leverages an API Gateway, Audit Log Redactor Workers, OpenAPI schemas for data redaction, and an Ingestor service to process and enrich logs. Key technical details include streaming HTTP requests via RPC, using OpenAPI schemas to identify and redact sensitive information, and a data pipeline involving Logfwdr, Logreceiver, and Buftee buffers. The Ingestor service transforms requests into standardized audit log records, and logs are enriched with user and authentication details before being stored. The system currently stores 30 days of logs, with plans to extend this to 18 months.
  43. Cloudflare incident on February 6, 2025This post details a significant incident where human error and insufficient validation safeguards during abuse remediation led to the disabling of the R2 Gateway service, impacting R2 object storage and several dependent services including Stream, Images, Cache Reserve, Vectorize, and Log Delivery. It highlights the critical role of R2 as a data infrastructure component and the cascading effects of its unavailability, while also emphasizing that no data was lost or corrupted. The incident underscores the importance of robust controls and validation in managing production data services.
  44. No hallucinations here: track the latest AI trends with expanded insights on Cloudflare RadarIntroduced a new dedicated "AI Insights" page on Cloudflare Radar, expanding on existing AI bot and crawler traffic graphs. The new page incorporates additional metrics to understand AI-related trends from multiple perspectives, including: relative popularity of publicly available Generative AI services based on 1.1.1.1 DNS resolver traffic, usage of robots.txt directives to restrict AI bot access to content, and open source model usage as seen by Cloudflare Workers AI. The AI Insights page provides visibility into AI bots and crawlers traffic trends, the popularity of Generative AI services, analysis of robots.txt files for AI user agents, and the popularity of models and tasks on Workers AI.
  45. Over 700 million events/second: How we make sense of too much dataThis post details Cloudflare's strategies for handling extremely high-volume data ingestion (706M events/sec) for its Analytics products. It introduces controlled downsampling techniques at various pipeline stages (Logfwdr, Logreceiver, Inserters) to retain information during component failures or overloads. Key technical contributions include: 1. Max-min fairness and weighted stream prioritization in Logfwdr's buffer management using a max-heap and goroutines. 2. Reservoir sampling-like 'bottomless buffers' that thin data automatically. 3. Adaptive sampling in Logreceiver based on customer size, pushing data at multiple resolutions. 4. The application of the Horvitz-Thompson estimator with Poisson sampling for deriving SUM, COUNT, and AVG from downsampled data, including variance estimation and confidence interval calculation. The post also discusses the mathematical underpinnings of these estimators and the challenges of calculating confidence intervals for averages when the total population size is unknown.
  46. The fall and rise of TikTok (traffic)This post analyzes the impact of the US ban on TikTok on DNS traffic, detailing the timeline of traffic drops and recovery. It also tracks the surge in DNS traffic for TikTok alternatives like RedNote (Xiaohongshu) in the US and globally, providing country-specific growth percentages and correlating these trends with app store rankings. The analysis leverages data from Cloudflare's 1.1.1.1 DNS resolver and autonomous system-level data.
  47. TikTok ban takes hold: data reveals sharp traffic decline and rapid shift to alternativesThis post details the real-time impact of the US TikTok ban on DNS and autonomous system traffic, showing significant drops for TikTok and surges for alternatives like RedNote. It also analyzes traffic shifts in other countries and the subsequent service restoration and partial recovery of TikTok traffic.
  48. Moving Baselime from AWS to Cloudflare: simpler architecture, improved performance, over 80% lower cloud costsBaselime migrated its observability data pipeline from AWS (Lambda, Kinesis, EC2 ClickHouse) to Cloudflare's Developer Platform. Key changes include replacing AWS Lambda with Cloudflare Workers for data reception and processing, utilizing JavaScript-native RPC for inter-component communication, and employing Durable Objects for stateful processing (real-time error tracking, log pattern detection) which replaces ElastiCache, Kinesis, and multiple Lambda functions. ClickHouse on EC2 was replaced by Workers Analytics Engine for high-cardinality analytics, with internal changes enabling high dimensionality and leveraging Adaptive Bit Rate (ABR) analytics for optimized query performance. Cloudflare Queues were adopted for asynchronous messaging, D1 for transactional data, and Workers KV for fast distributed storage. The migration followed a 'strangler fig pattern' to ensure uptime and data consistency.
  49. Impact of Verizon's September 30 outage on Internet trafficThis post details the analysis of a Verizon mobile network outage on September 30, 2024, using Cloudflare Radar data. It quantifies the impact on HTTP request volume from AS6167 (Verizon's mobile network ASN), showing a decline of up to 9% below expected levels. The analysis also breaks down traffic impact by specific impacted cities, highlighting significant drops in Omaha (approx. 30%) and other cities (10-20%), while noting some cities like New York and Atlanta showed week-over-week increases. The post concludes that voice services may have been more impacted than data services, based on the observed traffic patterns.
  50. Network trends and natural language: Cloudflare Radar’s new Data Explorer & AI AssistantIntroduced the Cloudflare Radar Data Explorer, a web-based interface for building complex API queries and visualizing results. Launched an AI Assistant that uses Cloudflare Workers AI to translate natural language questions into Radar API calls, with results displayed in the Data Explorer. Detailed the technical implementation of the AI Assistant using prompt engineering and multiple inference calls to LLMs, and the Data Explorer's visualization logic based on API response content.
  51. Removing uncertainty through "what-if" capacity planningThis post introduces 'Scenario Planner', a new system that enables sophisticated "what-if" capacity planning for Cloudflare's global network. It models future infrastructure needs by simulating various demand (customer growth, traffic shifts) and supply (data center failures, server additions) scenarios, translating them into a common unit of 'CPU Time'. The system outputs heatmaps and expected failover views to guide proactive infrastructure decisions, ensuring continued service availability and performance.
  52. A good day to trie-hard: saving compute 1% at a timeThis post introduces the 'trie-hard' open-source Rust crate, developed to optimize the `clear_internal_headers` function within Cloudflare's pingora-origin service. This optimization significantly reduces CPU utilization by over 1.28% by employing a custom trie data structure for efficient header removal, demonstrating a focus on micro-optimizations within core network infrastructure to improve overall performance and handle increased traffic.
  53. A global assessment of third-party connection tamperingThis post introduces new capabilities for detecting and analyzing global connection tampering. It details the methodology for identifying connection tampering signatures by analyzing abrupt connection closures and protocol anomalies. The post presents findings on the prevalence of connection tampering across different countries, correlating these findings with existing research and identifying potential causes like censorship and zero-rating. It also announces a new dashboard and API on Cloudflare Radar for near real-time visibility into connection tampering events.
  54. Bringing insights into TCP resets and timeouts to Cloudflare RadarIntroduced a new dashboard and API endpoint on Cloudflare Radar to track and analyze anomalous TCP connections (resets or timeouts within the first 10 ingress packets). This involved developing detection mechanisms to classify these connections and provide insights into network issues, scanning, and attacks. The post details the classification of connection stages (Post-SYN, Post-ACK, Post-PSH, Later) and provides guidance on interpreting the data.
  55. Introducing HTTP request traffic insights on Cloudflare RadarThis post introduces HTTP request traffic as a new metric on Cloudflare Radar, complementing existing byte-based traffic graphs. It details the implementation of this new metric, including its integration into the Overview and Traffic sections of Radar, and explains the value of request-based insights for distinguishing real events from data pipeline issues and for better reflecting user activity, especially with the rise of API traffic. The post also clarifies the difference between bytes and requests, using Portugal's traffic patterns as an example.
  56. A recent spate of Internet disruptionsThis post details the analysis of widespread Internet disruptions in Bangladesh, Syria, and Pakistan using Cloudflare Radar. It showcases the tracking of traffic anomalies, IP address space changes, and internet quality metrics during these events, providing specific examples of how Radar data was used to understand the scope and impact of government-imposed shutdowns and accidental infrastructure failures.
  57. Exploring Internet traffic during the 2024 U.S. Republican National ConventionThis post details the analysis of internet traffic and cyberattacks surrounding the 2024 U.S. Republican National Convention and the attempted assassination of Donald Trump. It presents data on DDoS attacks targeting political websites, traffic surges to news outlets following the assassination attempt, and specific traffic increases to Republican party and fundraising websites during the RNC. It also highlights a DDoS attack against a think tank during the RNC. The post emphasizes the short duration of most DDoS attacks and the need for automated mitigation systems. It also points to Cloudflare Radar and its 2024 Elections Insights report for further trend analysis.
  58. French elections: political cyber attacks and Internet traffic shiftsThis post analyzes internet traffic shifts and cyberattack trends during the 2024 French legislative elections. It details DDoS attacks targeting political parties, including peak request rates and total requests. It also examines internet traffic drops correlated with election result announcements, regional traffic variations, and changes in mobile device usage share. Additionally, it analyzes DNS trends for news outlets, TV domains, and social media platforms during the election periods.
  59. How the UEFA Euro 2024 football games are impacting local Internet trafficThis post details how Cloudflare's global network data is used to analyze the impact of major events like the UEFA Euro 2024 football tournament on local Internet traffic, observing drops in traffic during games and spikes in social media usage during breaks. It also highlights observed DDoS attacks during the event, including specific metrics like requests per hour and requests per second for a significant attack on a translation tool.
  60. Exploring the 2024 EU Election: Internet traffic trends and cybersecurity insightsThis post details the analysis of internet traffic trends and cybersecurity insights observed during the 2024 European Parliament election. It includes data on traffic drops during voting hours, increases as results were announced, and specific domain traffic growth for election-related sites. It also highlights government-focused cyberattacks observed during the election period, specifically mentioning DDoS attacks on Dutch political websites.
  61. Internet insights on 2024 elections in the Netherlands, South Africa, Iceland, India, and MexicoThis post details the analysis of internet traffic patterns and cyberattack trends during national elections in the Netherlands, South Africa, Iceland, India, and Mexico. It observes traffic dips during polling hours and spikes when results are announced, and notes DDoS attacks targeting political websites. Specific examples include DDoS attacks on Dutch political websites during the European Parliament election, traffic drops and spikes during Mexico's general election, and 44 days of traffic dips and mobile spikes during India's general election. The post also highlights the use of Cloudflare Radar for tracking these events and provides links to related reports and previous election analyses.
  62. An Internet traffic analysis during Iran's April 13, 2024, attack on IsraelThis post details the analysis of internet traffic patterns in Israel and Palestinian territories during the April 13, 2024, Iranian attack. It observes traffic surges in Israel correlated with the attack and a drop in traffic in Palestinian territories. It also analyzes traffic patterns in Iran, linking early morning declines to the conclusion of Ramadan and noting a subsequent increase. The post also reports on application layer attacks, noting a lack of significant changes targeting Israel and a historical trend of increased attacks on the Government Administration sector.
  63. Data Anywhere with Pipelines, Event Notifications, and WorkflowsThis post announces three new data-related services: Event Notifications for R2, which allows R2 bucket changes to trigger events in Cloudflare Queues; Pipelines, an upcoming streaming ingestion service for batching and writing data to R2; and Workflows, a durable execution engine for building reliable, event-driven applications on Cloudflare Workers. It details the architecture and use cases for each, including examples of event-driven workflows, data ingestion with Pipelines, and building AI-powered workflows with Workflows.
  64. R2 adds event notifications, support for migrations from Google Cloud Storage, and an infrequent access storage tierThis post introduces three new features for Cloudflare R2: Event Notifications (open beta) which allow triggering Cloudflare Workers via messages sent to a queue when R2 data changes; Super Slurper for Google Cloud Storage, enabling data migration from GCS to R2; and Infrequent Access storage class (private beta) for cost savings on less frequently accessed data, with an example of direct upload and mention of future object lifecycle policy automation.
  65. Log Explorer: monitor security events without third-party storageIntroduced Log Explorer, a new feature that allows users to investigate HTTP and Security Event logs directly within the Cloudflare Dashboard. Log Explorer is built on Cloudflare R2 and leverages the Delta Lake protocol for ACID-compliant storage and querying. It provides a SQL interface for analyzing raw log data, enabling detailed security investigations and reducing the need for third-party log storage solutions. This enhances the capabilities of Security Analytics by providing granular log access.
  66. Introducing Foundations - our open source Rust service foundation libraryThis post introduces 'Foundations,' an open-source Rust library designed to simplify the development of distributed, production-grade systems. It addresses key challenges in observability (logging, tracing, metrics), configuration, and security, drawing from Cloudflare's experience with its Oxy proxy framework. The library offers modularity, API ergonomics, and simplified setup, with specific enhancements in tracing (sampling flexibility, distributed trace stitching, trace forking) and logging (implicit context propagation). It also integrates with the Prometheus client for metrics and provides a procedural macro for easier metric definition.
  67. Cyber Week: analyzing Internet traffic and e-commerce trendsThis post details Cloudflare's analysis of Internet traffic and e-commerce trends during Cyber Week (Black Friday and Cyber Monday). It provides specific metrics on peak HTTP requests per second, daily request volumes, and DNS query volumes for both the 1.1.1.1 resolver and authoritative DNS servers. It also breaks down traffic by country and device type (mobile vs. desktop), and analyzes DNS trends for e-commerce sites, including category-specific insights (electronics, fast fashion, second-hand). Finally, it touches upon cyber threat trends, specifically DDoS attacks during this period.
  68. Internet traffic patterns in Israel and Palestine following the October 2023 attacksThis post details the analysis of internet traffic patterns and cyberattacks in Israel and Palestine following the October 2023 attacks. It quantifies traffic surges in Israel and network outages and traffic drops in Palestine, including specific Autonomous System (AS) numbers affected. It also analyzes the scale and timing of DDoS attacks targeting both regions, providing metrics like requests per second and daily blocked requests. The post highlights the use of Cloudflare Radar data to monitor these events and provides links to real-time data for continued observation.
  69. Sippy helps you avoid egress fees while incrementally migrating data from S3 to R2Introduced Sippy, an incremental migration service for R2 that copies data from S3 to R2 as it's requested, reducing egress fees by leveraging application requests. Sippy reduces migration stress and vendor lock-in by copying objects to R2 when they are requested from S3 and not found in R2. It can be used in conjunction with Super Slurper to complete the migration.
  70. Traffic anomalies and notifications with Cloudflare RadarThis post introduces the integration of an internal traffic anomaly detection and alerting tool into Cloudflare Radar's Outage Center (CROC). It details how country and network-level traffic anomalies are now displayed on CROC and made available via API. Additionally, new Radar notification functionality is launched, allowing users to subscribe to alerts for traffic anomalies, confirmed Internet outages, route leaks, or route hijacks at country or autonomous system levels, leveraging the Cloudflare dashboard's existing notification system. The post also provides technical details on the traffic anomaly detection methodologies and includes example API requests and responses for traffic anomalies.
  71. Gone offline: how Cloudflare Radar detects Internet outagesThis post details the technical approach to detecting Internet outages and traffic anomalies for Cloudflare Radar. It introduces a new system for automatically identifying anomalous traffic events (primarily traffic drops) across countries and Autonomous Systems (ASes). The system uses time-series analysis of various data sources (DNS, HTTP, NetFlows, NEL) to detect deviations from expected patterns. It outlines the process of verifying these anomalies, categorizing them as 'Verified' or 'False Positive', and the challenges of modeling diverse traffic patterns, including seasonality, trends, and external events like holidays. The post also discusses the criteria for selecting entities for anomaly detection based on traffic signal strength.
  72. Cloudflare Radar’s 2023 overview of new tools and insightsThis post details the new features and insights introduced to Cloudflare Radar in 2023, including the URL Scanner, Internet Quality page, Trending Domains, and Routing page. It also highlights general Internet insights observed throughout the year, such as the impact of the war in Ukraine, the Virgin Media outage, holiday trends, and the effects of conflicts in Sudan and Niger on Internet patterns, as well as the coronation of King Charles III.
  73. Cloudflare Workers database integration with UpstashThis post introduces the integration of Cloudflare Workers with Upstash databases (Redis, Kafka, QStash) via an OAuth2 flow. It demonstrates how to set up and use Upstash Redis within a Worker to fetch localized greetings based on the 'cf-ipcountry' header. It also suggests performance improvements using Workers Smart Placement or Upstash Global Database.
  74. Routing information now on Cloudflare RadarIntroduced a new 'Routing' page on Cloudflare Radar, providing detailed monitoring of BGP messages and related internet infrastructure data. This includes statistics on ASes, prefixes, and RPKI validation, as well as anomaly detection for BGP route leaks, hijacks, and RPKI invalid MOAS events. The page also offers insights into network connectivity (upstream, downstream, peering) and IP prefix origination. The data is made available through a developer API.
  75. Measuring the Internet's pulse: trending domains now on Cloudflare RadarIntroduced 'Trending Domains' to Cloudflare Radar, providing 'Trending Today' and 'Trending This Week' lists. This feature leverages aggregated 1.1.1.1 resolver data and the existing Radar Domain Rankings to identify domains with increasing popularity. The methodology involves analyzing rank volatility across different list sizes and comparing daily ranks against historical data (best rank of previous four days for 'Trending Today', weighted average for 'Trending This Week') to quantify popularity surges. This enhances the granularity of internet traffic insights.
  76. Introducing the Cloudflare Radar Internet Quality PageIntroduced the Cloudflare Radar Internet Quality page, which provides country and network-level insights into Internet connection performance (bandwidth) and quality (latency, jitter) over time. This page leverages aggregated results from the speed.cloudflare.com speed test tool and measurements against a set of Cloudflare & third-party targets (Internet Quality Index - IQI). The post details the methodology for calculating IQI and Connection Quality, including the use of trimeans and population weighting, and explains how to navigate the new page.
  77. Exam-related Internet shutdowns in Iraq and Algeria put connectivity to the testThis post details the use of Cloudflare Radar to analyze internet shutdowns in Iraq and Algeria for exam purposes. It showcases how Radar data (traffic, BGP announcements, 1.1.1.1 resolver usage) can be used to identify and characterize these disruptions, including differentiating between routing-based shutdowns and content blocking, and observing regional variations in impact. The post also highlights the integration of interactive graphs directly into the blog post using a new Cloudflare Radar feature.
  78. D1: We turned it up to 11This post announces a major update to Cloudflare D1, its native serverless SQL database. Key contributions include: significant performance and scalability improvements with a new storage backend (up to 20x faster reads, 6.8-11x faster writes), enhanced developer experience with a new console interface, JSON function support, and location hints. It also outlines the pricing model, including an always-free tier and usage-based billing, and introduces 'Time Travel' for point-in-time recovery (last 30 days). This post marks a significant maturation of D1 from alpha to a more robust and performant offering.
  79. Announcing database integrations: a few clicks to connect to Neon, PlanetScale and Supabase on WorkersIntroduces 'Database Integrations' for Cloudflare Workers, allowing seamless connection to third-party databases (Neon, PlanetScale, Supabase) via HTTP and outbound TCP support. Features an 'Integrations' tab in the dashboard for discoverability and an OAuth2 flow for secure credential management, storing them as encrypted environment variables. Mentions upcoming support for more databases and an integrations platform for third-party providers. Also notes upcoming Wrangler CLI support and account-level environment variables.
  80. The S3 to R2 Super Slurper is now Generally AvailableAnnounces the General Availability of R2 Super Slurper for one-time data migrations from S3 to R2. Highlights improvements in speed (up to 5x increase in objects copied per second), reliability (robust retries, chunked transfers for larger objects), and ease of use. Introduces the concept of incremental migration and opens a waitlist for its private beta. Provides a step-by-step guide for performing a one-time migration.
  81. Use Snowflake with R2 to extend your global data lakeThis post announces a partnership between Cloudflare R2 and Snowflake, enabling users to query data stored in R2 from Snowflake and load data from R2 into Snowflake. It provides step-by-step instructions for generating R2 API tokens and creating external stages in Snowflake using R2 credentials. It also demonstrates how to use the `COPY INTO` command for loading data and how to create external tables in Snowflake to query data in R2.
  82. Query Cloudflare Radar and our docs using ChatGPT pluginsIntroduced two new Cloudflare ChatGPT plugins: one for Cloudflare Radar, enabling natural language queries of Internet patterns via an enriched OpenAPI schema and itty-router-openapi, and another for Cloudflare Docs, using KV as a vector store for up-to-date documentation retrieval and Cron Triggers for index updates. This extends the capabilities of Cloudflare Radar by providing a new interface for data exploration and enhances developer experience by allowing natural language access to documentation.
  83. Cloudflare’s view of Internet disruptions in PakistanThis post details the impact of internet disruptions in Pakistan, analyzing traffic data at national, administrative unit, and city levels. It observes shifts in traffic sources (mobile vs. desktop) and the increased use of Cloudflare's 1.1.1.1 resolver during content blocking. The analysis includes specific Autonomous System Numbers (ASNs) for mobile and fixed broadband providers, and metrics on traffic volume, bot vs. human traffic, and latency.
  84. Introducing Object Lifecycle Management for Cloudflare R2Introduced Object Lifecycle Management for Cloudflare R2, enabling users to define rules for object expiration and aborting incomplete multipart uploads. This feature helps reduce storage costs by automatically deleting outdated logs, temporary files, or unfinished uploads. The functionality is accessible via the Cloudflare dashboard and the S3-compatible API.
  85. How we built Network Analytics v2This post introduces Network Analytics v2, a fundamental redesign of the backend systems providing real-time network layer traffic visibility for Magic Transit and Spectrum customers. It addresses the limitations of the previous version by enabling the reporting of stateful mitigation systems (like flowtrackd) and adopting principles from traditional network observability (Netflow/sFlow). The new design allows individual software components to emit metadata-rich packet samples, creating a more flexible and future-proof observability system. Key technical challenges in modifying existing systems like xdpd and l4drop to emit these samples are discussed.
  86. Effects of the conflict in Sudan on Internet patternsThis post details the analysis of Internet traffic patterns in Sudan during an armed conflict, showing significant drops in traffic, ISP outages, and shifts in messaging and social media platform usage, correlating these observations with real-world events and providing insights into the impact on connectivity. It leverages Cloudflare Radar data to illustrate these disruptions.
  87. Measuring network quality to better understand the end-user experienceIntroduced the Aggregated Internet Measurement (AIM) initiative, a new open format for displaying Internet quality. AIM aims to bridge the gap between traditional speed test metrics (bandwidth, packet loss, latency, jitter) and end-user experience by correlating these metrics with specific use cases. AIM also aims to help diagnose the location of network problems (e.g., WiFi vs. ISP) and provides a publicly available repository for data analysis. The source code for AIM and its score calculations is now open-sourced.
  88. Introducing Cloudflare’s new Network Analytics dashboardIntroduced a new Network Analytics dashboard for Magic Transit and Spectrum customers, featuring a refactored network-layer data logging pipeline. The dashboard provides enhanced visibility into traffic behavior, firewall events, and DDoS attacks, with detailed mitigation analytics, geographical accuracy based on data center ingestion, and packet sampling. It enables faster DDoS response by allowing users to pattern traffic and create Magic Firewall rules directly from the interface, and supports report generation.
  89. Internet disruptions overview for Q1 2023This post details observed Internet disruptions across Q1 2023, categorizing them by cause (government-directed, cable cuts, power outages) and providing specific examples with traffic data and geographical context. It contributes to the ongoing effort to monitor and report on global internet resilience by providing a quarterly overview of observed disruptions.
  90. Analyze any URL safely using the Cloudflare Radar URL ScannerIntroduced the Radar URL Scanner, a new free tool that compiles detailed technical reports for any given URL. The scanner leverages Cloudflare's Workers Browser Rendering API for headless scans and provides data across Security, Cookies, Network, Technology, DOM, and Performance categories. The post details the types of information available in each category and outlines future planned features such as API endpoints, private scans, and security recommendations.
  91. How Cloudflare runs Prometheus at scaleThis post details how Cloudflare operates a large-scale Prometheus deployment (916 instances, 4.9 billion time series) for network monitoring. It explains the concepts of metrics, labels, cardinality, samples, and time series, and discusses the challenges of high cardinality and memory consumption in Prometheus. It also outlines the initial steps in the Prometheus lifecycle: HTTP scrape and TSDB storage.
  92. One year of war in Ukraine: Internet trends, attacks, and resilienceThis post analyzes the impact of the war in Ukraine on internet trends, detailing traffic shifts, disruptions, and cyber attack patterns. It provides insights into network resilience, the effects of infrastructure damage, and the role of technologies like Starlink. The analysis leverages Cloudflare's network data to track changes in internet traffic volume and patterns across different regions and time periods, correlating them with geopolitical events and infrastructure attacks.
  93. Get notified about the most relevant events with Advanced HTTP AlertsThis post introduces Advanced HTTP Alerts, significantly enhancing Cloudflare's observability and analytics capabilities. Previously, HTTP alerts were generic. This update allows customers to customize alerts based on specific criteria like origin/edge response status codes, client IP addresses, and specific zones. This provides much finer-grained control for monitoring traffic anomalies and ensuring Service Level Objectives (SLOs) are met, directly contributing to the evolution of data infrastructure and analytics by providing more actionable insights from network traffic.
  94. Intelligent, automatic restarts for unhealthy Kafka consumersThis post details an advancement in Cloudflare's data infrastructure by introducing intelligent health checks for Kafka consumers. It addresses the challenges of ensuring application health in distributed systems like Kubernetes, particularly for Kafka consumers. The post explains the limitations of traditional health checks and proposes a new approach focused on message ingestion and offset commits to detect and automatically resolve unhealthy consumer states, thereby reducing incidents and manual intervention.
  95. Introducing Digital Experience MonitoringThis post introduces Cloudflare's new product, Digital Experience Monitoring (DEM). DEM aims to provide organizations with visibility into end-user connectivity and performance issues, addressing the 'finger-pointing' problem between IT teams. Key features include Zero Trust Fleet Status, Synthetic Application Monitoring, and Network Path Visualization. This expands Cloudflare's data and analytics offerings by providing granular insights into the digital experience of users within Zero Trust deployments, leveraging Cloudflare's extensive network data.
  96. Cloudflare Radar 2022 Year in ReviewThis post introduces the Cloudflare Radar 2022 Year in Review, presenting aggregated data on global Internet traffic growth (23% year-over-year), traffic trends influenced by global events (Olympics, Ukraine conflict, Hurricane Fiona), and category popularity (Technology, Business & Economy, Shopping & Auctions). It details methodology for traffic analysis and highlights regional variations in traffic distribution, such as in South Korea, Turkey, and Armenia. The post also discusses the impact of power outages on internet connectivity in Ukraine and Puerto Rico.
  97. Partnering with civil society to track Internet shutdowns with Radar Alerts and APIThis post details the launch of Radar Internet shutdown alerts and API access, enabling civil society organizations to track and document Internet disruptions. It highlights the collaboration with organizations like Access Now, Internet Society, and OONI, and introduces the Radar Outage Center (CROC).
  98. Send Cloudflare Workers logs to a destination of your choice with Workers Trace Events LogpushThis post introduces Workers Trace Events Logpush, a new capability that allows developers to send detailed logs from Cloudflare Workers to external object storage or analytics platforms. This extends existing debugging tools like `wrangler tail` by providing a historical record and enabling integration with popular observability tools, enhancing the overall observability of applications built on Cloudflare Workers.
  99. How Cloudflare instruments services using Workers Analytics EngineThis post details how Cloudflare uses its own Workers Analytics Engine to instrument its SQL API. It explains the need for observability and product insights, the process of logging events from the SQL API into Analytics Engine, and how to query this data for insights into query response times and customer behavior. It highlights the use of Adaptive Bit Rate (ABR) for handling large datasets and the ability to identify and prioritize missing features based on aggregated error types and specific customer issues. The post also discusses future improvements to the Grafana plugin and data retention.
  100. How we built it: the technology behind Cloudflare Radar 2.0This post details the engineering behind Radar 2.0, highlighting its architecture built on Cloudflare Pages with Functions, Remix for server-side rendering, and a Python/FastAPI backend API. It explains the use of Cloudflare Workers for the frontend API, transforming backend GraphQL data into a public REST API. Performance improvements through Early Hints and Lighthouse score enhancements are also discussed.
  101. Making static sites dynamic with Cloudflare D1This post introduces Cloudflare D1, a serverless SQL database service that integrates with Cloudflare Workers. It details how D1 enables developers to add dynamic, relational data to applications, specifically demonstrating its use for adding comments to a static blog site. The post covers setting up a D1 database, configuring bindings in wrangler.toml, executing SQL commands, and building API endpoints for creating and retrieving comments. It highlights D1's ability to augment existing applications and websites with dynamic data, bridging the gap between static site generators and fully dynamic applications.
  102. Migrate from S3 easily with the R2 Super SlurperIntroduced the R2 Super Slurper, a feature to migrate data from S3-compatible object storage to Cloudflare R2. The initial release supports one-time bulk imports via a UI and API, with future plans for incremental, on-demand copying ('sip by sip' migration). The 'sip by sip' migration strategy involves serving objects from R2 after they are requested from the origin, copying them to R2 for subsequent requests, and thereby saving on egress fees.
  103. Indexing millions of HTTP requests using Durable ObjectsIntroduced the R2 Log Retrieval API, built using Cloudflare Workers and Durable Objects, to index and query millions of customer logs stored in R2. This system uses Durable Objects for maintaining forward-indexes of RayIDs per log batch and the Streams API to process compressed log data without OOM errors. The indexing process involves decompressing, decoding, splitting, and collecting RayIDs, which are then stored in Durable Object storage. Log retrieval optimizes searches by leveraging the timestamp encoded in RayIDs and batch names to narrow down the search space, streaming relevant batches from R2 and filtering for matching records.
  104. Store and process your Cloudflare Logs... with CloudflareIntroduced Cloudflare Logs Engine, a new product for storing and processing Cloudflare Logs within Cloudflare. This enables customers to retrieve logs by time range and unique identifier (RayID beta) from R2 storage. The product aims to separate storage and compute costs for affordability and integrates with analytics to narrow down search queries. Future plans include ingesting logs from all plan types, supporting arbitrary filtering, and cross-dataset queries.
  105. Build applications of any size on Cloudflare with the Queues open betaThis post introduces Cloudflare Queues, a new open beta service that brings message queuing capabilities to the Workers Developer Platform. It enables developers to build more complex applications by decoupling components and buffering/batching calls to downstream services. The post details how to enroll in the beta, create a queue, and provides a practical example of using Queues with Workers and R2 for log aggregation, showcasing how Queues can improve efficiency and reduce costs.
  106. How the Brazilian Presidential elections affected Internet trafficThis post details the analysis of internet traffic patterns in Brazil during the 2022 Presidential elections using Cloudflare Radar data. It examines overall traffic trends, mobile device usage, and interest in election-related websites, candidate domains, and news organizations, correlating these with election events and official results. The analysis highlights how election days saw a decrease in daytime internet traffic, an increase in mobile usage, and significant spikes in DNS queries for election-specific content, with varying patterns for different categories of websites and candidates.
  107. Internet disruptions overview for Q3 2022This post details the observation and analysis of Internet disruptions during Q3 2022, categorizing them by cause (e.g., government-directed shutdowns) and geography, and leveraging Cloudflare Radar and internal tools to present traffic graphs and insights into these events. It highlights specific instances in Iraq, Cuba, Afghanistan, Sierra Leone, Somaliland, India, and Iran, providing traffic graphs and network-level details for each.
  108. Don't roll your own high cardinality analytics, use Workers Analytics EngineIntroduces Workers Analytics Engine, a new service for developers to store and analyze high-cardinality, time-series data generated by Cloudflare Workers. Highlights its use in powering Instant Logs by tracking request rates across sessions and data centers. Explains the underlying ABR technology for fast, interactive queries and the use of `_sample_interval` for reconstructing original event counts. Details improvements made for the open beta, including enhanced error messaging and comprehensive documentation. Mentions future plans for alerts, named fields, and a dashboard UX similar to Grafana.
  109. Goodbye, Alexa. Hello, Cloudflare Radar Domain RankingsThis post introduces Radar Domain Rankings, a new dataset based on aggregated 1.1.1.1 resolver data. It details the methodology for defining domain popularity (user population accessing a domain per unit of time, not just traffic volume) and the technical approach using machine learning models to predict domain ranks. It also explains the definition of a 'domain' for ranking purposes and the privacy considerations of using 1.1.1.1 data.
  110. The home page for Internet insights: Cloudflare Radar 2.0This post introduces Cloudflare Radar 2.0, a significant redesign of the platform for accessing and visualizing internet insights. Key technical changes include a new technology stack with GraphQL data endpoints and a public API, the website running on Cloudflare Pages and Workers with server-side rendering using Remix, adoption of SVG and a reusable data visualization component system, and visual TDD with Cosmos. The redesign focuses on improving ease of finding insights, expanding the number of insights (e.g., email), and enhancing sharing capabilities with downloadable/embeddable graphs and an API. The underlying data infrastructure, which powers Radar, continues to evolve as described in the 'Data Infrastructure & Analytics' thread.
  111. The status page the Internet needs: Cloudflare Radar Outage CenterLaunched the Cloudflare Radar Outage Center (CROC) as part of Radar 2.0. CROC serves as an archive of Internet outage information, including location, ASN, type, scope, cause, start time, and end time. It is built on top of the Radar API, making outage data accessible programmatically. The CROC interface includes a global map and a table for viewing outage details. Future plans include increased automation of outage detection and monitoring of cloud platform providers.
  112. Monitor your own network with free network flow analytics from CloudflareThis post introduces a free, early-access version of Magic Network Monitoring (formerly Flow Based Monitoring), a self-serve network flow analytics tool. It allows users to visualize traffic, filter by packet characteristics, and set volumetric alerts. The post highlights its utility for network engineers and managers to understand network traffic, identify peak volumes, detect malicious activity like DDoS attacks, and integrate with Magic Transit for mitigation.
  113. D1: our quest to simplify databasesThis post introduces D1, Cloudflare's first SQL database designed to work seamlessly with Cloudflare Workers. It details the development journey, user experience with Wrangler CLI and dashboard, local development and testing capabilities, automatic backup and restore features, and a sneak peek into the implementation of JavaScript transactions using a new `db.transaction()` API within stored procedures executed via Workers. This significantly expands Cloudflare's data storage offerings beyond key-value stores, Durable Objects, and blob storage, enabling full-stack applications on Cloudflare's global network.
  114. Logpush: now lower cost and with more visibilityIntroduced filtering capabilities for Logpush jobs, allowing customers to specify criteria for log delivery to reduce data volume and cost. Added alerting for failing Logpush jobs, notifying users when jobs have been retrying and failing for 24 hours. Launched a GraphQL API endpoint (`logpushHealthAdaptiveGroups`) to provide analytics on Logpush job health, including metrics like bytes pushed, records pushed, and push status.
  115. Protests spur Internet disruptions in IranThis post details how Cloudflare Radar data was used to analyze and report on Internet disruptions in Iran during protests. It specifically shows the impact on traffic volumes for various Iranian ISPs (TCI, Iran Mobile Communications Company, RighTel, MTN Irancell) and regional/national traffic trends. It also analyzes DNS blocking by examining requests to Cloudflare's resolver for social media platforms (Instagram, WhatsApp) and the blocking of DNS-over-HTTPS/TLS (DoH/DoT) to Cloudflare's resolver. The analysis demonstrates the use of Cloudflare's network data to monitor and understand censorship and network interference.
  116. Regional Services comes to India, Japan and AustraliaThis post announces the general availability of Regional Services in India, Japan, and Australia, expanding the capability to control data processing locations to the Asia Pacific region. It reiterates the functionality of Regional Services, which allows customers to specify data centers that can decrypt and inspect traffic, thereby adhering to data localization regulations and contractual obligations. The post also explains the technical implementation of Regional Services, which works by forwarding raw, encrypted traffic to data centers within the customer-selected region for decryption and application of Layer 7 products.
  117. Store and retrieve your logs on R2This post introduces the capability to store and retrieve Cloudflare logs directly on Cloudflare R2 object storage. Previously, logs could only be exported to third-party destinations. This new feature provides a cost-effective solution, eliminating egress fees and offering S3 API compatibility for existing tooling, thereby centralizing storage and retrieval for easier access and analysis.
  118. R2 is now Generally AvailableThis post announces the General Availability of R2, Cloudflare's object storage service. It highlights key features such as S3 compatibility, automatic region selection, deep integration with Cloudflare Workers for custom logic and API creation, presigned URLs for delegated access, public bucket support for seamless integration with other Cloudflare features, and transparent pricing with a generous free tier. The post also outlines future plans including object lifecycles, jurisdictional restrictions, and live migration without downtime. The technical contribution lies in the detailed explanation and code examples demonstrating R2's capabilities and its integration with Workers, showcasing how developers can build applications leveraging R2 for data storage and access.
  119. Using Cloudflare R2 as an apt/yum repositoryThis post details how Cloudflare leverages its R2 object storage service to host apt/yum repositories for its cloudflared daemon. It explains the mechanics of package management systems like apt-get and yum, and outlines the process of creating a deb/rpm file, structuring the repository files, uploading them to R2, and serving them via a Cloudflare Worker. This demonstrates an innovative application of object storage for software distribution, enhancing reliability and reducing operational overhead.
  120. Log analytics using ClickHouseThis post details the migration of Cloudflare's error logging pipeline from Elasticsearch to ClickHouse. It outlines the challenges faced with Elasticsearch, including mapping explosion, poor multi-tenancy support, operational overhead, and garbage collection issues, especially at Cloudflare's scale of 35-45 million HTTP requests per second with 500K-800K errors per second. The solution involves adopting ClickHouse, a column-oriented database, to improve query performance, reduce storage costs, and enable the storage of all error logs without sampling. Key ClickHouse features leveraged include its column-oriented storage for fast sequential scans, design for analytical workloads with many columns, sparse indexing, efficient LZ4 compression with configurable codecs (e.g., Double-Delta for DateTime, Gorilla for Float, LowCardinality for String), and linear scalability. An efficient inserter using Cap'n Proto messages and optimized batch sizes is also described.
  121. Performance isolation in a multi-tenant database environmentThis post details the challenges and solutions for performance isolation in Cloudflare's multi-tenant Postgres clusters. It describes the problem of tenants contending for shared resources (CPU, memory, disk I/O, database connections) leading to throughput and latency degradation. The post outlines previous manual solutions like connection limits and introduces a new solution involving gateway concurrency throttling at the PgBouncer layer, including runtime commands and open-source contributions. It also explores future solutions like congestion avoidance inspired by TCP Vegas and tenant resource quotas to proactively manage performance and prevent resource starvation.
  122. Open sourcing our fork of PgBouncerThis post contributes to the data infrastructure evolution by detailing improvements made to PgBouncer, a critical component for managing Postgres connection pooling. Specifically, it addresses an authentication bug that impacted HBA authentication and introduces new features for enforcing and dynamically adjusting per-user connection pool limits and per-connection pool limits at runtime. By open-sourcing these enhancements, Cloudflare not only improves its own database infrastructure but also benefits the broader open-source community, aligning with the theme of leveraging and contributing to foundational technologies.
  123. When the window is not fully open, your TCP stack is doing more than you thinkThis post delves into the complex memory and window management of the TCP receiving side within the Linux kernel. It clarifies how Linux manages TCP receive buffers and windows, answering questions about the actual capacity of receive buffers and the speed at which they can be filled. The post explains the concepts of skmem_rb, recv-q, skmem_r, and the advertised window, detailing how settings like tcp_adv_win_scale influence performance. It highlights the critical issue of exceeding memory budgets, leading to packet drops and performance degradation, and discusses the roles of TCP Coalesce and TCP Collapse in mitigating these problems.
  124. Using Apache Kafka to process 1 trillion inter-service messagesThis post details Cloudflare's extensive adoption and internal tooling development around Apache Kafka for inter-service communication. It highlights the use of Kafka to process over a trillion messages, the creation of internal tools like Messagebus-Client and a connector framework for easier adoption, the implementation of strict schemas using Protobuf for forward/backward compatibility, and the establishment of robust observability through automated metrics and alerts. It also provides a practical example of how this framework supports the Alert Notification System.
  125. A story about AF_XDP, network namespaces and a cookieThis post details a debugging journey and fix for a crash in flowtrackd related to AF_XDP's lack of network namespace awareness. It highlights the use of AF_XDP for high-performance packet processing in DDoS mitigation, the challenges encountered with shared UMEM features across network namespaces, and the identification of a bug related to interface index numbers. The post also contributes to the understanding of low-level network infrastructure optimizations and their impact on security services.
  126. Internet disruptions overview for Q2 2022This post details the analysis of Internet disruptions observed by Cloudflare during Q2 2022, categorized by cause (physical events like fiber cuts, and intentional government shutdowns). It presents traffic graphs from Cloudflare Radar and internal tools to illustrate the impact of these events on specific networks (Comcast, Telkom SA, CANTV, AAE-1, SMW-5, CityWest) and countries (Syria, Sudan, Algeria). The analysis includes observations on asymmetric shutdowns, DNS retry floods, and the impact of content blocking versus nationwide shutdowns.
  127. Exam time means Internet disruptions in Syria, Sudan and AlgeriaThis post details the use of Cloudflare Radar to track and analyze nationwide Internet shutdowns in Syria, Sudan, and Algeria during exam periods. It presents data showing traffic drops to zero, analyzes asymmetric inbound/outbound traffic patterns in Syria, and highlights the shift from full shutdowns to targeted website/application blocking in Algeria, supported by Network Error Logging (NEL) data. The post also references previous work on exam-related shutdowns in Sudan and the economic impact of such disruptions.
  128. Monitoring our monitoring: how we validate our Prometheus alert rulesThis post details Cloudflare's use of Prometheus as its core monitoring system and introduces a new open-source tool developed to validate Prometheus alert rules. It explains the potential pitfalls of alert rule creation, such as typos or incorrect query logic, leading to silent failures where alerts don't fire when they should. The post elaborates on Prometheus querying basics (instant vs. range queries) and how these can lead to unexpected empty results, impacting alert reliability. The contribution lies in addressing the critical aspect of ensuring the accuracy and trustworthiness of the monitoring system itself, thereby improving the overall observability platform.
  129. Integrating Network Analytics Logs with your SIEM dashboardIntroduces Network Analytics Logs for Magic Transit, Magic Firewall, and Spectrum customers on the Enterprise plan. These logs provide packet samples of traffic dropped and passed by specific mitigation systems, enabling near real-time visibility into network traffic and DDoS attacks. The post details how to set up Logpush jobs to feed these logs into SIEM systems like Splunk, including API token creation, Splunk HEC setup, and the Logpush job configuration. It also highlights the cost-saving benefits of storing logs in R2.
  130. Introducing Workers Analytics EngineIntroduced Workers Analytics Engine, a new serverless time-series analytics platform built on Cloudflare's existing analytics technology. It allows developers to collect and query telemetry data from Cloudflare Workers using a structured event log model with blobs (strings for grouping/filtering) and doubles (numbers for aggregation). Key features include low-latency reporting (seconds), unlimited cardinality of label values, fast queries across any timescale, and integration with SQL and GraphQL APIs for data querying. It also supports visualization with tools like Grafana. The initial implementation requires configuring bindings in wrangler.toml and using the `writeDataPoint` API within Workers. Future plans include named blobs/doubles, on-the-fly binding definition, and richer dashboard analytics.
  131. Logs on R2: slash your logging costsThis post introduces the integration of Cloudflare's Logpush service with R2 object storage, allowing enterprise customers to store raw logs at a significantly reduced cost compared to traditional cloud object storage providers. It highlights the cost savings and introduces plans for enhanced log retrieval capabilities, including a Log Retrieval API beta.
  132. Watching Eurovision 2022 on Cloudflare RadarThis post details the use of Cloudflare Radar to track internet traffic trends related to the Eurovision Song Contest. It showcases the application of DNS name resolution data to estimate traffic to video platforms and fan sites, and social media trends. The post highlights the ability to visualize internet traffic aggregates by country and by website type, demonstrating Cloudflare's data analysis capabilities for real-time event monitoring.
  133. Workers visibility: announcing Logpush for Worker’s Trace EventsThis post announces the upcoming integration of Workers execution logs into Logpush for Enterprise customers. This new capability will provide unstructured console.log messages, exceptions, and request/response metadata, enabling granular debugging, performance analysis, and the exposure of logs to end-users for SaaS platforms built on Workers. It complements existing visibility tools like wrangler tail and Workers Analytics Engine.
  134. PIPEFAIL: How a missing shell option slowed Cloudflare downThis post details a critical incident where a missing shell option ('pipefail') in a Kubernetes cron job led to corrupted configuration data in Quicksilver. This corruption caused the `dosd` service to fail, which in turn blocked requests in the Front Line service for up to five seconds per request, causing a global slowdown. The incident highlights the importance of robust error handling in data pipelines and the cascading effects of seemingly minor configuration oversights on critical infrastructure.
  135. Cloudflare Radar’s new ASN pagesThis post introduces new Autonomous System (ASN) pages to Cloudflare Radar. These pages provide detailed traffic statistics, protocol usage, and security information for individual ASNs. They also display geographical traffic distribution and the volume of BGP announcements, offering insights into network changes and disruptions. The feature allows users to deep dive into any ASN by plugging its ASN into the Radar URL.
  136. Cloudflare ObservabilityThis post introduces Cloudflare's future vision for observability, defining it as having three core components: monitoring, analytics, and forensics. It details how Cloudflare plans to enhance each of these areas: expanding notification types for monitoring, making the dashboard more customizable for analytics (especially for security policy validation), improving log access and storage (including on R2), and introducing tracing to provide end-to-end request lifecycle visibility. The post emphasizes the goal of providing a single pane of glass for all network activity and a unified experience across security, performance, and developer products.
  137. Get full observability into your Cloudflare logs with New RelicThis post announces a new direct integration between Cloudflare's Logpush service and New Relic, a third-party observability platform. This integration aims to provide customers with enhanced end-to-end visibility by allowing them to correlate Cloudflare logs with data from their applications and origin servers within New Relic One. The partnership eliminates the need for middleware, leading to faster log delivery and reduced costs for mutual customers, and includes a quickstart guide for easy setup.
  138. Internet traffic patterns in Ukraine since February 21, 2022This post details the analysis of internet traffic patterns in Ukraine from February 21, 2022, onwards, using Cloudflare's network data. It showcases how traffic data from various cities (Kyiv, Lviv, Uzhhorod, Ternopil, Rivne, Kharkiv, Sumy, Izyum, Donetsk, Mariupol, Osypenko, Irpin, Bucha, Enerhodar, Severodonetsk) can be used to infer real-world events such as population displacement and the impact of cyberattacks. It also analyzes cyberattack traffic, including DDoS attacks and layer 7 attacks against .ua domains, demonstrating Cloudflare's mitigation efforts and the types of attacks observed.
  139. Slicing and Dicing Instant Logs: Real-time Insights on the Command LineThis post introduces the ability to consume Cloudflare Instant Logs via WebSockets and analyze them in real-time on the command line using tools like Websocat and Angle Grinder. It details how to create an Instant Logs session, specify desired log fields (e.g., ClientIP, FirewallMatchesActions), set sample rates, and apply filters based on request path. A practical example demonstrates how to verify a newly deployed firewall rule by analyzing logs for requests to a specific path and filtering by country and firewall actions.
  140. Missing Manuals - io_uring worker poolThis post delves into the intricacies of `io_uring`, a high-performance asynchronous I/O API in Linux, specifically focusing on its worker pool management. It clarifies how `io_uring` classifies I/O requests into bounded and unbounded work, and how it manages thread creation for each category. The post details how to monitor and control the unbounded worker pool size, which is crucial for network I/O operations that are central to Cloudflare's infrastructure. It provides practical examples using in-kernel tracing to observe `io_uring`'s behavior and offers solutions for developers to effectively utilize this powerful I/O runtime.
  141. Landscape of API TrafficThis post quantifies the growth of API traffic on Cloudflare's network in 2021, showing it now represents 54% of total requests and is growing twice as fast as traditional web traffic. It details API growth by industry, highlighting significant increases in Banking, Retail, and Financial Services. The post also analyzes API characteristics, noting POST and GET as the dominant methods (98% of requests) and JSON as the primary payload encoding (97% of requests). It introduces Cloudflare's API Shield products (API Discovery, Schema Validation, mTLS, API Abuse Detection) as tools to protect against API-specific threats.
  142. Internet shut down in Kazakhstan amid unrestThis post details Cloudflare Radar's observation and analysis of the nationwide internet shutdown in Kazakhstan in January 2022. It uses Radar data to show traffic drops across various ASNs, including mobile services, and correlates these with BGP updates. The post also tracks intermittent restorations of internet services, linking them to public announcements by the Kazakh President. It highlights the impact on businesses and Bitcoin mining, and compares the event to other internet shutdowns.
  143. From 0 to 20 billion - How We Built Crawler HintsThis post details the engineering behind Cloudflare's Crawler Hints product, which aims to reduce the environmental impact of web searches by informing search indexers about content changes. It explains how cache miss data from Cloudflare's global network is processed, utilizing Kafka for message buffering and Redis for aggregation and deduplication. The post also covers the integration with search partners like Yandex and Bing through the IndexNow API and highlights the importance of customer opt-in, monitoring, and system resilience in rolling out such a large-scale feature.
  144. Sanitizing Cloudflare Logs to protect customers from the Log4j vulnerabilityThis post addresses a critical security vulnerability (Log4j) by introducing a log sanitization feature for Cloudflare Logs. Customers can now opt-in to automatically redact potentially exploitable tokens (`${`) within their logs via Logpush jobs, protecting them from remote code execution attempts when consuming logs with vulnerable software. This enhances the security and integrity of the data infrastructure analytics pipeline.
  145. What’s new with Notifications?This post details significant enhancements to Cloudflare's Notification center. It highlights the importance of timely notifications for customers to be aware of issues impacting their internet properties. The post announces the shift from primarily email-based notifications to a greater emphasis on webhooks, with expanded support for platforms like DataDog, Discord, OpsGenie, and Splunk. Crucially, it introduces Notification History, allowing users to review past notifications for up to 90 days (depending on plan), addressing the issue of lost or missed alerts.
  146. Store your Cloudflare logs on R2This post announces the upcoming ability for customers to store Cloudflare logs directly on Cloudflare R2 storage. It highlights the cost-effectiveness, ease of setup, long-term retention, and product-agnostic nature of this new logging solution. It contrasts this with traditional logging methods and existing Cloudflare logging products like Instant Logs and Logpush, positioning R2 storage as a solution for retroactive log inspection and analysis.
  147. Attack Maps now available on RadarIntroduced Radar Maps with geographical distribution of application-level attacks (origin and target) and Sankey diagrams for attack flows. These visualizations are built using Cloudflare Workers and Workers KV.
  148. Sudan was cut off from the Internet for 25 daysThis post details the restoration of internet traffic in Sudan after a 25-day shutdown, using Cloudflare Radar data to illustrate the recovery patterns. It highlights the role of specific ISPs (Mobitel, MTN, Sudatel) in the restoration and notes the impact on mobile traffic, which is dominant in Sudan. The post also contextualizes the event by referencing previous internet shutdowns in the country and emphasizes the ongoing utility of Cloudflare Radar for monitoring global internet traffic and disruptions.
  149. Sudan: seven days without Internet access (and counting)This post details the use of Cloudflare Radar to monitor and report on a country-wide internet outage in Sudan. It highlights the flatlining of internet traffic, the minimal blips of activity, and provides historical context of previous internet shutdowns in the region. The post demonstrates Radar's capability to track and visualize internet access disruptions at a national level.
  150. Sudan woke up without InternetThis post details the use of Cloudflare Radar to track and visualize Internet traffic disruptions in Sudan during a political event. It presents data showing a significant drop in overall Internet traffic, HTTP requests, and traffic across all major ASNs. The post also highlights the impact on both mobile and desktop traffic and notes a change in Layer 3&4 DDoS attack patterns. The data is presented through screenshots of Radar's interface, demonstrating its capability to monitor global and country-specific Internet health.
  151. “Look, Ma, no probes!” — Characterizing CDNs’ latencies with passive measurementThis post introduces a novel method for passively measuring and characterizing CDN latencies. It addresses the limitations of active measurements (like RIPE Atlas and RUM services) due to probe distribution, data validation issues, and privacy concerns. The post details a two-step process: first, predicting anycast catchments by compiling network footprints (ISPs, data centers, IXPs, BGP data) to infer which CDN data center a client request will reach; and second, predicting CDN path latencies by leveraging Cloudflare's own passive RTT measurements as a proxy. This contributes a new analytical capability to understand and compare CDN performance without direct, active probing of competitors.
  152. Multi-User IP Address DetectionThis post introduces a new capability within Cloudflare's data analytics to detect multi-user IP addresses. It explains the challenges posed by shared IP addresses due to NAT and CG-NAT, and details the methods developed to identify these multi-user IP prefixes by analyzing distinct source ports and user agents. This enhances Cloudflare's security tools by providing more granular traffic identification.
  153. Announcing Cloudflare R2 Storage: Rapid and Reliable Object Storage, minus the egress feesThis post announces Cloudflare R2 Storage, a new object storage service. R2 offers S3 API compatibility, zero-cost egress bandwidth, and competitive pricing for storage and operations. It features automatic data tiering for performance and cost optimization, a migration service from S3-compatible storage, and plans for multi-region storage with jurisdictional controls. R2 is designed for high durability (eleven 9s) and reliability, with redundancy across regions. It integrates deeply with Cloudflare Workers for dynamic data transformation and manipulation, enabling use cases like IoT data streaming and CDN asset storage.
  154. Unboxing the Last Mile: Introducing Last Mile InsightsThis post introduces 'Last Mile Insights', a new feature that provides customers with visibility into connection failures occurring between their end-users and Cloudflare properties. It explains the concept of the 'Last Mile' in internet connectivity, details how Last Mile Insights works using Network Error Logging (NEL) and machine learning, and presents a case study with Canva to illustrate its value in diagnosing and resolving ISP-related connectivity issues. The post also reiterates Cloudflare's commitment to global network expansion to minimize Last Mile impact.
  155. How we built Instant LogsThis post introduces 'Instant Logs,' a new real-time log access feature built on Cloudflare Workers and Durable Objects. It addresses the need for immediate visibility into network events, complementing the existing Logpush product by providing sub-second latency through edge-based processing, filtering, and sampling (Reservoir Sampling) to handle varying traffic volumes.
  156. Data at Cloudflare just got a lot faster: Announcing Live-updating Analytics and Instant LogsIntroduced Live-updating Analytics for Pro, Business, and Enterprise customers, providing real-time visibility into data as it arrives. Launched Instant Logs for Enterprise customers, enabling near real-time viewing of HTTP request logs in the dashboard with an average delay of two seconds. This significantly reduces the 'glass-to-glass' time for log data, enabling faster response to security threats and operational issues.
  157. Introducing: Custom Hostname AnalyticsThis post introduces 'Custom Hostname Analytics,' a feature that allows SaaS providers to offer their end-customers visibility into traffic analytics for their specific domains hosted on the SaaS platform. This extends Cloudflare's existing analytics capabilities to provide granular insights for individual customer hostnames, enabling SaaS providers to better understand customer usage, billing, and infrastructure scaling, and to demonstrate the value of Cloudflare's performance and security benefits to their own clients.
  158. Introducing logs from the dashboard for Cloudflare WorkersIntroduced integrated logging and exception tracking for Cloudflare Workers, accessible directly from the dashboard and via the `wrangler tail` CLI. This feature eliminates the need for manual logging setup by developers, automatically capturing `console.log` output and exceptions. The dashboard provides a viewable stream of logs and exceptions, with filtering by event status and type. The `wrangler tail` command offers advanced filtering by IP address, status, HTTP method, sampling rate, and generic search queries, with options for pretty and JSON output formats. This enhancement aims to simplify debugging and improve the understanding of Worker execution.
  159. More products, more partners, and a new look for Cloudflare LogsIntroduced new datasets for Cloudflare Logs: Firewall Events and Network Error Logging (NEL) Reports. Expanded Logpush destinations to include any S3-compatible API. Enabled direct integrations for pushing logs to Microsoft Azure Sentinel, Splunk, Sumo Logic, and Datadog via the UI. Redesigned the Logpush UI for selecting datasets and destinations.
  160. Sudan's exam-related Internet shutdownsThis post details the use of Cloudflare Radar to track Internet traffic patterns in Sudan, specifically identifying and analyzing Internet shutdowns implemented by the government during exam periods. It highlights the ability to observe traffic drops, differentiate between mobile and desktop access during these shutdowns, and provides a direct link to the country-specific Radar page for real-time monitoring. The post also mentions the potential for future shutdowns and the use of Twitter for assistance in tracking them.
  161. Syria’s exam-related Internet shutdownsThis post details the use of Cloudflare Radar to track and visualize internet shutdowns in Syria during exam periods. It highlights how Radar's country-specific pages and traffic data can be used to observe significant drops in internet usage corresponding to scheduled outages, providing concrete evidence of these events.
  162. Improving your monitoring setup by integrating Cloudflare’s analytics data into Prometheus and GrafanaThis post details the implementation of a custom Prometheus exporter by Labyrinth Labs that pulls data from Cloudflare's GraphQL Analytics API. The exporter transforms Cloudflare analytics data (requests, bandwidth, cache utilization, threats, SSL usage, HTTP response codes, traffic origin by country/location, and data center performance) into Prometheus metrics. It includes design considerations (Go, API SDK, Goroutines for parallel requests), deployment instructions (Docker image, Helm chart for Kubernetes), and visualization guidance using Grafana dashboards. Future work includes integrating more analytics data like firewall, DoS, and network analytics.
  163. Announcing Cloudflare’s Database PartnersThis post announces Cloudflare's strategic partnerships with Macrometa and Fauna to enhance state management at the edge for Cloudflare Workers. It introduces Macrometa as a globally replicated NoSQL database with integrated search, pub/sub, and stream processing capabilities, and Fauna as a global transactional database delivered as a secure, cloud API. The post highlights how these partnerships enable developers to build stateful applications with low-latency data access, querying, and strong consistency, complementing existing Workers KV and Durable Objects by addressing use cases like complex data querying and integration with existing systems of record. Case studies for a bookstore demo app with Macrometa and the MeetKai AI assistant with Fauna are provided to showcase the benefits.
  164. Moving k8s communication to gRPCThis post details Cloudflare's migration of internal service-to-service communication from REST to gRPC within their Kubernetes environment. It highlights the benefits of gRPC for microservice architecture, including improved performance through protobuf serialization and HTTP/2 multiplexing, enhanced developer usability by abstracting network calls into function calls, and better security through Kubernetes NetworkPolicies. The post also discusses the challenges faced with REST and Kafka for large data transfers like DNS zones and how gRPC addresses these.
  165. Enhancing privacy-focused Web Analytics to better meet your metrics needsThis post introduces several enhancements to Cloudflare's Web Analytics platform. It adds support for measuring multiple websites per account (up to 10), with the ability to view combined analytics across all sites using tags. Crucially, it introduces support for Single-Page Applications (SPAs) by tracking changes to the History API (pushState and onpopstate events) to capture route changes. Additionally, Core Web Vitals are now available within Web Analytics for all customers, providing insights into website performance.
  166. Lessons Learned from Scaling Up Cloudflare’s Anomaly Detection PlatformThis post details the evolution and scaling of Cloudflare's Anomaly Detection platform, a key component of its Bot Management system. It highlights the platform's technical implementation using HBOS, ClickHouse, and Redis, and chronicles its journey from a single monolithic service struggling with 10k RPS to a microservices-based architecture handling nearly 500k RPS. Key contributions include optimizations in Redis (PFMERGE, recency register, key encoding) and the strategic shift to microservices to reduce load on shared dependencies like ClickHouse and improve independent scalability of pipeline components.
  167. Conntrack turns a blind eye to dropped SYNsThis post delves into the intricacies of the Linux kernel's conntrack module and its interaction with the Netfilter framework. It explains why conntrack doesn't log dropped SYN packets by default, how to enable its hooks, and uses BPF tracing to demonstrate the flow of packets through Netfilter. This contributes to the understanding of data infrastructure by providing a deep dive into a fundamental network component that impacts traffic analysis and observability.
  168. Automating data center expansions with AirflowThis post introduces Cloudflare's Provisioning-as-a-Service (PraaS) platform, built using Apache Airflow, to automate the complex process of data center expansions and server provisioning. It details how manual Standard Operating Procedures (SOPs) were transformed into API-driven tasks within Airflow DAGs, integrating with tools like Salt, Prometheus, and JIRA. The post highlights features like failure handling, logging, notifications, Jinja templating, sensors for preconditions and human intervention, branching, multi-DAGs, and strategies for creating scalable DAGs to manage over 200 data centers, resulting in a 90% reduction in manual operational tasks.
  169. Uganda's January 13, 2021 Internet Shut DownThis post details the analysis of the January 2021 internet shutdown in Uganda using Cloudflare Radar data. It demonstrates how traffic patterns, application-level attacks, and BGP routing statistics from Cloudflare's network and the Uganda Internet eXchange point were used to monitor and report on the outage and its subsequent restoration. The post highlights the ability to visualize significant drops and returns to near-zero internet traffic, providing concrete data on the impact of government-ordered internet suspensions.
  170. Soar: Simulation for Observability, reliAbility, and secuRityThis post introduces SOAR (Simulation for Observability, Reliability, and Security), a new system designed to address the challenges of testing complex, homogeneous edge environments. SOAR creates dedicated data centers for running simulations with synthesized customer traffic and applications, mirroring production software stacks. It enhances isolation and coordination between simulations, improves engineer-friendliness through an internal coordinator service, and allows for more complex, interactive simulations, particularly for products like Magic Transit, ensuring reliability and security before production deployment.
  171. Internet traffic disruption caused by the Christmas Day bombing in NashvilleThis post analyzes the impact of the Christmas Day bombing in Nashville on internet traffic flow, specifically focusing on the disruption to AT&T services and its effect on Cloudflare traffic. It presents traffic flow data before, during, and after the outage, illustrating the duration and recovery of services. The analysis demonstrates the use of traffic flow data to understand real-world network disruptions.
  172. Computing Euclidean distance on 144 dimensionsThis post details the engineering challenges and solutions for computing Euclidean distance on high-dimensional data (144 dimensions) for an image matching algorithm used in Cloudflare's CSAM scanning tool. It explores naive brute-force approaches, the limitations of SIMD (AVX2) due to memory bandwidth, and the failure of space partitioning algorithms like VP-trees due to the curse of dimensionality. The post highlights a breakthrough with a 'short distance' variation that computes only a subset of dimensions initially, and further optimization by leveraging the application's characteristic of expecting few matches (proving non-existence of neighbors within a threshold) to achieve significant performance gains.
  173. Cloudflare’s privacy-first Web Analytics is now available for everyoneThis post announces the public availability of Cloudflare Web Analytics, allowing users to integrate it into any website via a JavaScript snippet. This expands the analytics offering beyond existing Cloudflare customers who previously relied on DNS changes. The post details the privacy-first approach, emphasizing no user tracking, no cookies, and no fingerprinting. It explains the core metrics (page views, visits) and features like global filters, zoom, and group-by functionality. The technical implementation leverages client-side tracking, similar to other popular analytics tools, contrasting with Cloudflare's traditional edge-based analytics.
  174. A Byzantine failure in the real worldThis post details a significant incident involving a Byzantine fault in the etcd cluster due to a partial switch failure, which cascaded into database primary promotion issues and impacted API and dashboard availability. It highlights the challenges of handling degraded states in redundant systems, the limitations of RAFT in Byzantine scenarios, and the subsequent improvements made to database cluster management, user session handling, and auto-remediation configuration. It also touches upon the ongoing research into Byzantine Fault Tolerance (BFT) for production systems.
  175. Moving Quicksilver into productionThis post details the production deployment of Quicksilver, Cloudflare's new distributed data store for configuration distribution, which replaces the older Kyoto Tycoon. It describes the "bridge" strategy for parallel operation, the gradual rollout process, and the technical challenges encountered, such as replication saturation due to bootstrapping, I/O errors from aging SSDs, write amplification issues with LMDB, and the need for more dynamic topology management. The post highlights significant performance improvements observed after migrating critical services like FL to Quicksilver.
  176. ClickHouse Capacity Estimation FrameworkThis post details the development of a capacity estimation framework for ClickHouse clusters. It describes the process of identifying relevant metrics (unreplicated data from system.parts), collecting historical data from Thanos using an aiohttp client, forecasting future disk usage with Facebook's Prophet, and visualizing real and predicted data in Grafana. The framework automates capacity planning, making it more transparent and accessible to product engineers.
  177. Diving into /proc/[pid]/memThis post details the implementation of the `/proc/[pid]/mem` file within gVisor's virtual file system. This was a crucial step to enable proper stack unwinding and debugging for processes running inside the sandbox, which previously resulted in unhelpful stack traces. The post dives into the Linux kernel's implementation of `/proc/[pid]/mem`, including access checks and read operations, and explains how these concepts were adapted for gVisor's sandboxing environment. This directly contributes to Cloudflare's data infrastructure and analytics by enhancing the diagnostic capabilities within their sandboxed environments.
  178. Introducing Cloudflare RadarThis post introduces Cloudflare Radar, a new public-facing service that exposes aggregated data on Internet traffic patterns, domain popularity, and cyberattack activity. It comprises three key components: Radar Internet Insights (news, trend data, domain popularity, attack activity, technology trends), Radar Domain Insights (detailed information on individual domains including popularity, security, and content categories), and Radar IP Insights (network and geographic information for IP addresses). This launch democratizes access to insights previously only available internally at Cloudflare.
  179. Free, privacy-first analytics for a better webIntroduced a new, privacy-first web analytics service accessible to all users via a JavaScript beacon, complementing existing edge analytics. This service defines 'visits' as a privacy-friendly measure of interaction, avoiding individual user tracking, client-side state (cookies, localStorage), and IP address fingerprinting. It offers features like drag-to-zoom and grouping for time-series charts and leverages ABR analytics technology for fast data serving. Plans include bot detection and removal for customers of Bot Management.
  180. Explaining Cloudflare's ABR AnalyticsThis post introduces Adaptive Bit Rate (ABR) analytics as a technique to improve the quality and responsiveness of Cloudflare's analytics products. ABR is applied to data stored in ClickHouse, where data is written at multiple resolutions (covering seven orders of magnitude) into separate tables. This approach enables consistent response times and a better user experience by calculating analytics at a resolution matching the query's needs, thus saving resources and allowing for ad hoc queries on reduced resolution tables. The post also discusses aggregations and rollups, and how ABR handles 'topK' and 'count-distinct' aggregations, improving resilience and scalability by adapting to system conditions and providing appropriate answers reliably within predictable timeframes.
  181. Start measuring Web Vitals with Browser InsightsThis post introduces the integration of Web Vitals (Largest Contentful Paint, First Input Delay, Cumulative Layout Shift) into Cloudflare's Browser Insights product. It explains the importance of Web Vitals for understanding user experience beyond traditional metrics like TTFB. The post highlights the benefits of using Cloudflare's Browser Insights for RUM data, including continuous updates, support for all browsers (not just Chromium), and its free availability. It details how to enable Browser Insights via the Speed tab in the dashboard and mentions the underlying JavaScript beacon mechanism.
  182. Using data science and machine learning for improved customer supportThis post introduces the application of data science and machine learning to improve customer support and identify attack traffic. It details three specific techniques: improving language classification by considering multiple signals beyond off-the-shelf algorithms, optimizing fuzzy string matching for error message extraction using Cosine similarity for better performance, and developing a multi-dimensional data analysis approach with unsupervised clustering for identifying complex attack patterns like credential stuffing.
  183. Health Check Analytics and how you can use itThis post introduces Standalone Health Check Analytics, a new feature that provides detailed logs and visualizations for diagnosing origin server issues. It enhances existing health check capabilities by offering error breakdown, availability graphs, response code mismatch details, and a request waterfall view to analyze TCP connection, TLS handshake, and time to first byte. It also highlights the global nature of health checks, showing regional latency variations.
  184. How we use HashiCorp NomadThis post details the adoption of HashiCorp Nomad for managing 'management services' in Cloudflare's edge data centers. This directly contributes to the reliability and operational efficiency of the data infrastructure that underpins analytics capabilities by ensuring critical services remain available and efficiently resourced, even in distributed environments. It highlights how Nomad's dynamic task scheduling addresses challenges in managing services that are not customer-facing but essential for data center operations, thereby indirectly supporting the overall data infrastructure and analytics ecosystem.
  185. Export logs from Cloudflare Gateway with LogpushIntroduced the ability to export logs from Cloudflare Gateway using the Cloudflare Logpush Service. This allows organizations to analyze and audit tens or hundreds of millions of DNS queries and security decisions by exporting data to third-party storage destinations or SIEM tools. The feature builds on the existing Logpush Service and provides configuration options for fields and storage destinations within the Cloudflare for Teams UI.
  186. Test your home network performanceIntroduced speed.cloudflare.com, a network performance testing tool that measures download, upload, latency, and jitter. The tool provides transparent methodology, allows users to download raw measurements, and runs entirely on Cloudflare Workers at the edge. The backend code is open-sourced on GitHub.
  187. Backblaze B2 and the S3 Compatible API on CloudflareThis post details the integration of Backblaze B2's S3-compatible API with Cloudflare Workers. It provides a practical example of how to use Cloudflare Workers and the aws4fetch library to sign requests to Backblaze B2, enabling direct serving of content from Backblaze B2 through Cloudflare's network. It also highlights the benefits of the Bandwidth Alliance in reducing first-mile bandwidth costs.
  188. CUBIC and HyStart++ Support in quicheThis post details the integration and testing of CUBIC and HyStart++ congestion control algorithms within Cloudflare's quiche (QUIC) implementation. It explains the concepts of QUIC congestion control, Reno, CUBIC, and HyStart++, and presents lab test results demonstrating that CUBIC and HyStart++ improve performance and significantly reduce packet loss compared to Reno, especially under packet loss conditions. This contributes to the ongoing optimization of Cloudflare's network transport layer for better performance and reliability.
  189. Stream Firewall Events directly to your SIEMThis post introduces the ability to stream Firewall Events directly to SIEMs using Cloudflare Logs and Logpush jobs. It details the process of setting up this integration using Terraform and Sumo Logic, including prerequisites, Sumo Logic collector and HTTP source creation, and Cloudflare API token generation. The key technical contribution is enabling customers to send only security-relevant events to their logging platforms, reducing data volume and improving the speed of event delivery (within 60 seconds).
  190. Cloudflare Dashboard and API Outage on April 15, 2020This post details a significant outage of the Cloudflare Dashboard and API due to a physical disconnection of critical fiber optic cables during planned maintenance. It highlights the importance of robust internal data infrastructure and control plane availability, the challenges of disaster recovery and failback, and the need for improved documentation, labeling, and process design to prevent single points of failure in physical infrastructure.
  191. Project Crossbow: Lessons from Refactoring a Large-Scale Internal ToolThis post details the refactoring and revamp of 'Crossbow', an internal tool used by Cloudflare's Technical Support Engineers for network diagnostics. It highlights the process of deprecating the UI to focus on the CLI, rearchitecting the pub/sub pipeline for better stability and security, and integrating JWT authentication with Cloudflare Access. The post also emphasizes the importance of risk management, feedback loops, and test-driven development in improving internal tooling, leading to a significant increase in usage and the consolidation of other diagnostic tools into Crossbow.
  192. Introducing Quicksilver: Configuration Distribution at Internet ScaleThis post introduces Quicksilver, a new system designed to replace Kyoto Tycoon for configuration distribution at internet scale. It details the limitations of the previous system, Kyoto Tycoon, particularly its performance degradation under heavy write loads due to exclusive write locks and issues with data synchronization and corruption. Quicksilver is presented as the solution to these problems, enabling faster and more reliable distribution of configuration changes across Cloudflare's global network.
  193. Announcing Network AnalyticsIntroduced Network Analytics dashboard for Magic Transit and BYOIP customers, providing packet-layer visibility into network and transport-layer traffic patterns and DDoS attacks. This extends the existing analytics platform to L3/L4, complementing the previous L7-focused dashboards. The Network Analytics dashboard offers near real-time visibility into traffic and attacks blocked at the edge, with data accessible via GraphQL. Data storage architecture provides one year of insights, with sampled IP flow logs and summarized attack logs. The post details customer use cases for incident response and reporting, highlighting key metrics for each.
  194. When Bloom filters don't bloomThis post introduces the use of Bloom filters as a probabilistic data structure for efficient large-scale data deduplication, specifically in the context of analyzing IP spoofing data. It details the development of a custom tool 'mmuniq-bloom' and uses profiling to identify performance bottlenecks related to random memory access and cache misses, demonstrating the practical challenges and trade-offs of using Bloom filters for massive datasets.
  195. Gen X Performance TuningThis post details Cloudflare's in-depth performance tuning efforts for their AMD EPYC 7642 processors in their Gen X servers. It explores the nuances of Thermal Design Power (TDP) and dynamic power, the impact of determinism modes (Performance vs. Power), and the benefits of Configurable TDP (cTDP). The post quantifies the gains achieved, including an additional 6% performance throughput from out-of-the-box tuning, with further improvements of 2% and 4% from power determinism and cTDP adjustments respectively. It also touches upon the Nodes Per Socket (NPS) configuration, noting no significant performance deltas observed.
  196. Impact of Cache LocalityThis post contributes to the data infrastructure and analytics thread by detailing an in-depth hardware performance analysis. It specifically evaluates the impact of cache locality on Cloudflare's workloads by comparing AMD 2nd Gen EPYC processors with Intel Xeon processors. The post highlights how advancements in CPU architecture, particularly larger L3 caches, can significantly improve request processing throughput and reduce cache miss rates, leading to more efficient server performance.
  197. An EPYC trip to Rome: AMD is Cloudflare's 10th-generation Edge server CPUThis post details Cloudflare's strategic decision to adopt AMD EPYC 7642 processors for its 10th-generation edge servers, replacing Intel Xeon Platinum 6162. The primary driver for this change is the pursuit of higher 'Requests per Watt,' a key metric for operational efficiency and cost management. The post provides a detailed comparison of the new AMD CPUs against the previous Intel generation, highlighting improvements in core density, power efficiency (TDP per core), memory channels, and PCIe lanes. Extensive benchmarking across cryptography, compression, and Go-specific operations demonstrates significant performance gains with the AMD EPYC processors, reinforcing their suitability for Cloudflare's compute-intensive workload.
  198. How we used our new GraphQL Analytics API to build Firewall AnalyticsThis post details the development of Firewall Analytics, the first dashboard product to utilize the new GraphQL Analytics API. It explains how the API enables quick prototyping and iteration by allowing precise data definition and response formatting. The post provides concrete GraphQL query examples for visualizing firewall event spikes over time, identifying top offending IPs and user agents, and drilling down into individual firewall events for false positive analysis. It highlights the use of `firewallEventsAdaptiveGroups` for aggregations and `firewallEventsAdaptive` for raw event data, demonstrating how these can be mapped to UI elements for a comprehensive security workflow.
  199. Introducing the GraphQL Analytics API: exactly the data you need, all in one placeIntroduced a new GraphQL Analytics API that provides a unified endpoint for accessing all Cloudflare analytics data, including performance, security, and reliability metrics. This API allows users to query specific data sets, fields, and time ranges, and supports introspection for discovering available data. It deprecates the existing Zone Analytics API.
  200. Introducing Load Balancing AnalyticsThis post introduces Load Balancing Analytics, a new feature that provides detailed insights into traffic steering decisions, origin health, and latency for Cloudflare Load Balancing. It highlights the traffic flow overview, a latency map, and the underlying GraphQL Analytics API, enabling customers to optimize their infrastructure, reduce costs, and increase application availability.
  201. Announcing deeper insights and new monitoring capabilities from Cloudflare AnalyticsThis post announces the general availability of a new API for Cloudflare Analytics, enabling customers to build custom dashboards and explore their data. It also highlights the expansion of Firewall Analytics to Business and Pro plans, the introduction of Load Balancing Analytics, and new monitoring tools for origin servers. The Account Analytics dashboard is released in beta for aggregated domain information.
  202. Supercharging Firewall Events for Self-ServeThis post introduces a completely overhauled Firewall Event log for Free, Pro, and Business customers. Key improvements include a smoother user experience without modals, expanded detail views for each event, an 'Additional matches' count to clarify rules and services triggered for a single request, and powerful freeform search capabilities across all visible fields within a Firewall Event. It also adds the ability to scope searches to specific date and time windows (24 hours for Free/Pro, 72 hours for Business) within the last two weeks.
  203. A gentle introduction to Linux Kernel fuzzingThis post introduces the application of coverage-guided fuzzing, specifically using AFL and KCOV, to the Linux Kernel's netlink machinery. It details how to set up and run this process within a KVM virtualization environment, demonstrating a novel approach to uncovering bugs in low-level kernel components. This contributes to the broader theme of data infrastructure by showcasing advanced techniques for ensuring the robustness and security of the underlying operating system components that Cloudflare relies upon.
  204. Get Cloudflare insights in your preferred analytics providerThis post announces partnerships with Chronicle Security, Datadog, Elastic, Looker, Splunk, and Sumo Logic to provide pre-built dashboards for analyzing Cloudflare logs and metrics. These dashboards are available as Cloudflare Apps and leverage Logpush to ingest data into cloud storage solutions like Google Cloud Storage and Amazon S3. The dashboards include over 90 panels covering performance, security, and reliability of websites and applications on Cloudflare's network.
  205. Cloudflare architecture and how BPF eats the worldThis post details Cloudflare's extensive use of eBPF (extended Berkeley Packet Filter) within its Linux-based edge network infrastructure. It highlights how eBPF is employed for critical functions such as volumetric DoS mitigation (XDP eBPF), layer 4 load balancing (XDP eBPF), and sophisticated socket filtering for UDP traffic. The post also touches upon custom kernel patches and iptables modules (TPROXY) used to overcome Linux networking limitations for advanced dispatching and handling of high volumes of traffic, showcasing a deep dive into kernel-level optimizations for network performance and security.
  206. eBPF can't count?!This post details a critical bug discovered in the eBPF verifier related to 64-bit arithmetic operations (subtraction) when run by unprivileged users. It explains how a security mitigation for Spectre variant 1 attacks inadvertently broke this functionality, leading to incorrect calculations. The post demonstrates the bug, explains the underlying cause by examining kernel code and eBPF bytecode, and discusses workarounds like using 32-bit ALU operations or waiting for kernel patches. This contributes to the understanding of how Cloudflare leverages and debugs low-level kernel technologies for its infrastructure.
  207. xdpcap: XDP Packet CaptureThis post introduces xdpcap, an open-source tool that acts as a replacement for tcpdump specifically for eXpress Data Path (XDP) programs. It enables packet capture and filtering at the kernel level, addressing the visibility gap created by XDP's early packet processing. The post details the technical implementation, including the use of cBPF to eBPF compilation via cbpfc and eBPF tail-calls for instrumentation, thereby enhancing the ability to analyze and debug network traffic at a very low level.
  208. Logpush: the Easy Way to Get Your Logs to Your Cloud StorageIntroduced Logpush, a new service for Enterprise customers that automatically pushes request logs to cloud storage providers (Amazon S3, Google Cloud Storage) instead of requiring customers to pull them via the Logpull API. This simplifies log management and enables easier integration with other infrastructure data for analytics and debugging.
  209. io_submit: The epoll alternative you've never heard aboutThis post introduces the Linux AIO API as a potential alternative to epoll for network servers, focusing on its ability to batch syscalls like read and write. It explains how Linux AIO, traditionally for disk I/O, can be used with network sockets to reduce context switching overhead and improve performance, especially in high-traffic scenarios. The post also details how to bypass the `io_getevents` syscall by directly interacting with the kernel's ring buffer for completion events, further optimizing performance.
  210. Logs from the EdgeThis post introduces an example Cloudflare Worker implementation that leverages sub requests to send traffic logs to an arbitrary location, specifically an Elastic stack. It details the Worker code for collecting log data (timestamp, URL, referrer, method, ray ID, IP, host, user agent, country code, colo, TLS version/cipher, status) and asynchronously posting it via a PUT request to an ELK endpoint. It also discusses how this enables real-time logging for all users, contrasting it with Enterprise Log Share (ELS) and suggesting improvements with Argo Tunnel and Access for securing the logging backend.
  211. Introducing ebpf_exporterIntroduced ebpf_exporter, a tool that leverages eBPF to collect low-level system metrics, specifically focusing on disk I/O latency histograms. This addresses the limitations of traditional Prometheus exporters like node_exporter and cAdvisor by providing detailed event-level data, enabling more accurate performance analysis and alerting.
  212. Tracing System CPU on Debian StretchThis post details a deep dive into system-level performance issues encountered during an upgrade from Debian Jessie to Debian Stretch, specifically focusing on increased system CPU usage and RCU stalls on Kafka nodes. It highlights the use of `perf` for profiling and `dmesg` for identifying kernel-level issues like memory allocation stalls and dropped kernel messages, ultimately leading to the identification of a regression in the Linux kernel between Debian Jessie and Stretch as the root cause.
  213. eBPF, Sockets, Hop Distance and manually writing eBPF assemblyThis post introduces the use of eBPF (extended Berkeley Packet Filter) and socket options (`SO_ATTACH_BPF`) for advanced network introspection. It details how eBPF can be used to extract IP TTL values from TCP connections to determine hop distance, a novel approach to verifying traffic routing outside of the datacenter. The post also delves into the technical challenges and implementation details of writing eBPF assembly, including the use of eBPF maps for data sharing between kernel and userspace.
  214. Data-driven development with Cloudflare Mobile SDKIntroduced the Cloudflare Mobile SDK for Android and iOS developers to visualize and understand their mobile app's network utilization. The SDK helps identify slowdowns caused by network errors, top N requests, slow requests, and third-party calls. It aims to correlate network experience data with existing engagement data tracked in tools like Mixpanel, Amplitude, and Heap. The SDK is free, has no cap on active users or metrics tracked, and does not require using Cloudflare's infrastructure. Privacy is a key consideration, with no collection of persistent identifiers and no selling of collected data.
  215. HTTP Analytics for 6M requests per second using ClickHouseThis post details the migration of Cloudflare's HTTP traffic analytics pipeline from a legacy PostgreSQL/Citus-based system to ClickHouse. The new pipeline addresses limitations of the old system, including single points of failure, complex codebase, and high maintenance costs. It leverages ClickHouse for its columnar storage, real-time analytical capabilities, horizontal scalability, and fault tolerance. The post discusses schema design for aggregated data, performance tuning of ClickHouse, and the integration of the Kafka table engine for direct ingestion.
  216. Squeezing the firehose: getting the most from Kafka compressionThis post details Cloudflare's experience with Kafka compression, specifically focusing on the improvements introduced in Kafka 0.11.0. It highlights the challenges faced with older versions and the benefits of the new batch compression strategy. The post includes performance testing of different compression codecs (gzip, lz4, snappy) on various data types (nginx errors, capnp-encoded requests) and analyzes the trade-offs between CPU, disk, and network usage. It also touches upon the integration of these improvements into their Go-based Kafka clients (sarama) and identifies potential bottlenecks in the compression process.
  217. Creating a single pane of glass for your multi-cloud Kubernetes workloads with CloudflareThis post introduces the application of Cloudflare Load Balancer to manage multi-cloud Kubernetes workloads, enabling intelligent traffic distribution, avoiding vendor lock-in, and improving cost-effectiveness. It highlights the ability to achieve a 'single pane of glass' for these distributed systems.
  218. Using Go as a scripting language in LinuxThis post introduces the capability of using Go as a scripting language in Linux by leveraging the `binfmt_misc` kernel module. It details the challenges with the traditional shebang approach for Go scripts and provides a practical solution using a custom interpreter (`gorun`) and kernel module registration to enable direct execution of `.go` files, improving prototyping and development efficiency.
  219. However improbable: The story of a processor bugThis post details the investigation and resolution of a mysterious processor bug affecting Intel Xeon E5-2650 v4 (Broadwell) processors. It highlights the process of identifying and debugging hardware-level issues that manifest as unexpected software crashes (SIGSEGV, SIGABRT, SIGILL) and the use of core dumps for post-mortem analysis. The discovery of the bug required meticulous examination of system logs and core dumps to rule out software bugs and eventually pinpoint the hardware as the source of the problem.
  220. SYN packet handling in the wildThis post delves into the low-level details of TCP SYN packet handling within the Linux kernel, specifically focusing on the SYN Queue and Accept Queue mechanisms. It explains how these queues function, their size limitations, the impact of application performance on connection handling, and the role of SYN cookies in mitigating SYN flood attacks. The post also highlights the use of tools like `ss` and custom SystemTap scripts for monitoring and debugging these network operations, contributing to Cloudflare's deep understanding and optimization of its network infrastructure.
  221. Technical reading from the Cloudflare blog for the holidaysThis post highlights several key developments in Cloudflare's data infrastructure and analytics capabilities. It details how Cloudflare analyzes 1 million DNS queries per second using a custom pipeline with ClickHouse and Grafana, and how they made their DNS stack 3x faster. It also touches on hardware performance comparisons (ARM vs. Intel), SystemTap for kernel analysis and performance optimization, and the analysis of DDoS attacks like WireX and reflection attacks. Additionally, it mentions the use of LavaRand for randomness generation and the impact of leap seconds on DNS.
  222. Go, don't collect my garbageThis post delves into performance optimization within Go programs, specifically addressing the impact of garbage collection (GC) on multi-core performance. It demonstrates how tuning the GOGC variable can drastically improve throughput for computationally intensive tasks by reducing the frequency and overhead of GC cycles, a critical consideration for high-performance systems.
  223. On the dangers of Intel's frequency scalingThis post details a performance issue observed with Intel's AVX-512 instruction set, specifically its impact on the ChaCha20-Poly1305 cipher. It highlights how dynamic frequency scaling, triggered by AVX-512 usage, can significantly reduce overall CPU performance, even when AVX-512 is used for a small portion of the workload. The post contrasts the performance of OpenSSL and BoringSSL implementations and suggests disabling AVX-512 for general-purpose workloads to avoid unintended throttling, contributing to the understanding of low-level system performance and optimization strategies.
  224. ARM Takes Wing: Qualcomm vs. Intel CPU comparisonThis post introduces the evaluation of ARM-based processors (Qualcomm Centriq) as a potential alternative to Intel CPUs for Cloudflare's server infrastructure. It details performance benchmarks for cryptography and compression, compares hardware specifications, and assesses the ecosystem readiness of ARM for Cloudflare's software stack (NGINX, LuaJIT, Go). This marks an exploration into diversifying hardware choices for performance and cost optimization.
  225. Perfect locality and three epic SystemTap scriptsThis post delves into the optimization of network packet processing at the kernel level, specifically exploring the REUSEPORT socket option and its evolution with SO_INCOMING_CPU and SO_ATTACH_REUSEPORT_CBPF. It demonstrates how these features can improve packet locality by ensuring packets are processed on the same CPU that received them, thereby enhancing CPU cache locality and reducing cross-CPU interrupts. The post also introduces three advanced SystemTap scripts to measure and verify these improvements, though it notes that direct end-to-end performance gains for high-level TCP applications were not significantly measurable.
  226. LavaRand in Production: The Nitty-Gritty Technical DetailsThis post details the technical implementation of LavaRand, a system that uses a wall of lava lamps as a source of true randomness for cryptographic operations. It explains the concepts of true vs. pseudorandomness, the importance of entropy pools in operating systems, and how LavaRand mixes physical entropy with local system entropy to create a more secure and unpredictable source for Cloudflare's production fleet, acting as an additional layer of defense against potential compromises of standard entropy sources or CSPRNGs.
  227. Randomness 101: LavaRand in ProductionThis post introduces LavaRand, a system that uses lava lamps as a secondary source of cryptographic randomness for Cloudflare's production servers. It explains the importance of unpredictable randomness in cryptography, how computers typically generate it, and how LavaRand leverages the unpredictable physical behavior of lava lamps, captured by a camera and fed into a CSPRNG, as a hedge against potential flaws in primary randomness sources. This adds a novel approach to data input for security-critical systems.
  228. Using Google Cloud Platform to Analyze Cloudflare LogsThis post describes an integration between Cloudflare's log sharing service and Google Cloud Platform (GCP) components (Storage, Cloud Function, BigQuery, Data Studio) to provide Enterprise customers with an end-to-end solution for analyzing their Cloudflare access logs. It details the data flow from log retrieval via REST API to storage in GCP Storage, import into BigQuery via Cloud Functions, and visualization using Data Studio. It also mentions the availability of a GCP credit for new accounts and directs users to GCP pricing for cost details.
  229. Why does one NGINX worker take all the load?This post delves into the intricacies of load balancing in NGINX, specifically addressing the problem of uneven distribution of incoming connections across worker processes. It explains the different models for handling accept() calls, the Linux kernel's differing load balancing behaviors for blocking accept() vs. epoll-and-accept, and introduces SO_REUSEPORT as a solution for achieving more even load distribution. The post also highlights the critical trade-off between even load balancing and latency, demonstrating how SO_REUSEPORT can sometimes lead to degraded latency distributions under high load.
  230. Three little tools: mmsum, mmwatch, mmhistogramThis post introduces three small, open-sourced command-line tools: `mmhistogram` for generating ASCII histograms from data, `mmwatch` for observing the rate of change in command output, and `mmsum` for summing lists of floating-point numbers. These tools provide developers with convenient ways to quickly analyze and visualize data directly from the command line, contributing to the operational analytics capabilities.
  231. A container identity bootstrapping toolThis post introduces PAL (Permissive Action Link), a tool developed by Cloudflare to securely distribute secrets (API tokens, private keys, passwords) to Dockerized production applications. It addresses the challenges of managing secrets in CI/CD workflows and containerized environments by providing a service identity bootstrapping mechanism. PAL allows encrypted secrets to be decrypted at runtime after a service's identity has been established, working with orchestration frameworks like Mesos and supporting encryption methods like PGP and Red October. The post highlights the importance of secure secret management for production systems and the decision to open-source PAL.
  232. How Cloudflare analyzes 1M DNS queries per secondThis post details the technical evolution of Cloudflare's DNS analytics pipeline. It describes the use of Cap'n Proto for edge log generation, Apache Kafka for buffering, and the adoption of ClickHouse as an OLAP system for storing and querying DNS telemetry data. Key technical details include the ingestion adapter service written in Go, ClickHouse's MergeTree storage engine, columnar storage, sparse indexing, and the careful selection of primary keys for optimized read performance. The post also discusses the challenges of aggregating high-cardinality data and the decision to use materialized views for predefined aggregations.
  233. How eero mesh WiFi routers connect to the cloudThis post introduces how eero, an IoT device manufacturer, leverages Cloudflare's CDN and high availability to provide a resilient and cost-effective method for its devices to check internet connectivity. This acts as a 'cloud canary' and helps eero monitor its devices' connection to the cloud without privacy concerns or reliance on third-party sites. It highlights Cloudflare's role in supporting the specific needs of Internet of Things products and ensuring scale across millions of devices.
  234. LuaJIT Hacking: Getting next() out of the NYI listThis post details efforts to optimize LuaJIT performance by addressing 'Not Yet Implemented' (NYI) features, specifically focusing on the `next()` function used for table iteration. It introduces a new tool called 'loom' for better visualization and analysis of LuaJIT's JIT compilation process, and demonstrates how to improve the compilation of `next()` to avoid performance penalties associated with interpreter fallback.
  235. Want to see your DNS analytics? We have a Grafana plugin for thatThis post introduces a Grafana plugin for Cloudflare DNS analytics, enabling users to visualize DNS traffic patterns, response codes (including NXDOMAIN increases during DDoS attacks), and query breakdowns by data center and type. It also highlights the underlying DNS analytics API and provides installation instructions for the plugin and example curl commands for API usage. Virtual DNS customers can also leverage a custom dashboard for traffic distribution and RTT from origins.
  236. How and why the leap second affected Cloudflare DNSThis post details a critical incident where a leap second caused a negative time value in Cloudflare's DNS resolution software (RRDNS), leading to a temporary outage for customers using CNAME records. It highlights a specific bug in the Go programming language's time handling and how Cloudflare's internal systems, particularly the weighted selection algorithm for upstream DNS resolvers, were affected. The post contributes to the understanding of Cloudflare's data infrastructure by showcasing the challenges of timekeeping in distributed systems and the engineering effort involved in identifying and fixing such subtle, yet impactful, issues.
  237. Using Guzzle and PHPUnit for REST API TestingThis post introduces the use of Guzzle and PHPUnit for automated REST API testing within a PHP environment. It details setting up the testing environment using Composer, configuring PHPUnit, and writing tests to simulate HTTP requests and validate responses. This contributes to the data infrastructure by providing a robust method for ensuring the quality and reliability of APIs, which are increasingly critical for modern internet services.
  238. So you want to expose Go on the InternetThis post details how to expose Go services directly to the internet by optimizing `crypto/tls` and `net/http` configurations. It provides specific guidance on setting secure TLS cipher suites, curve preferences, and implementing crucial timeouts (`ReadTimeout`, `WriteTimeout`, `IdleTimeout`) to ensure stability and prevent resource exhaustion. It also covers HTTP/2 considerations and best practices for `ServeMux` usage and logging, marking a shift from relying solely on reverse proxies to directly exposing Go applications with enhanced security and performance.
  239. This is strictly a violation of the TCP specificationThis post details the debugging of a rare 522 error, tracing it back to a subtle issue where an application leaking sockets in the CLOSE_WAIT state on the loopback interface caused connection timeouts for new connections. It highlights how this application-level problem, not a network issue, can lead to symptoms that appear to violate TCP specifications, impacting Cloudflare's edge connectivity.
  240. CloudFlare's JSON-powered Documentation GeneratorThis post introduces Cloudflare's use of JSON Schema and JSON Hyper-Schema for defining their RESTful API. It details the development of a toolchain, including `json-schema-loader`, `json-schema-example-loader`, and `doca`, to automatically generate HTML documentation from these schemas. This demonstrates an evolution in how Cloudflare manages and exposes its data and API, contributing to internal tooling and developer experience.
  241. More data, more dataThis post details the state of processing edge logs (HTTP and DNS) for Cloudflare's Data Team. It highlights the scale of data generated (360TB/day), the infrastructure used (Kafka clusters, CitusDB), and what has worked well (log forwarder, Kafka, Log Share, CitusDB, Platform/SRE support). It also outlines future work, including improving customer-facing service reliability, building new analytics systems (mentioning Spark Streaming), developing new data pipelines, and enhancing tooling for complex analysis.
  242. Why we use the Linux kernel's TCP stackThis post delves into Cloudflare's strategic decision to utilize the Linux kernel's TCP stack, explaining the benefits of hardware independence, time-sharing, and running multiple network applications. It highlights the trade-offs of userspace networking and kernel bypass, particularly the loss of concurrent application support. The post details Cloudflare's use of 'partial kernel bypass' for DDoS mitigation, offloading iptables to a userspace process to avoid IRQ storms. It also emphasizes the value of the Linux TCP stack's features, debugging tools, and ecosystem over custom solutions.
  243. Go coverage with external testsThis post introduces a technique for capturing Go test coverage data from end-to-end tests, which is not natively supported by the Go toolchain. It details a workaround involving a dummy test that executes the main function and compiles a binary with coverage enabled. The post also explains how to merge this coverage data with unit test coverage using `gocovmerge` for a more complete picture of code execution, specifically mentioning its application in the RRDNS project.
  244. Partial kernel bypass merged into netmap mainThis post details the integration of Cloudflare's work on netmap into the mainline netmap repository. The key contribution is the development of a more configurable netmap mode that allows for partial kernel bypass, enabling specific network queues to be detached from the host stack while others remain attached. This allows for more flexible packet processing and the ability to capture transmitted packets by moving them to the RX host ring. New flags (`NR_TX_RINGS_ONLY`, `NR_RX_RINGS_ONLY`) were introduced for finer control over ring pair usage.
  245. The story of one latency spikeThis post details the investigation and resolution of rare, significant latency spikes experienced by a customer. It showcases the use of advanced debugging tools like System Tap and flame graphs to pinpoint the root cause within the Linux kernel's `tcp_collapse` function, which was exacerbated by large TCP receive buffer sizes. The post demonstrates how tuning the `net.ipv4.tcp_rmem` sysctl parameter significantly improved network performance and stability, highlighting Cloudflare's commitment to deep system-level performance optimization.
  246. Single RX queue kernel bypass in Netmap for high packet rate networkingThis post details Cloudflare's contribution to the Netmap project, introducing a 'single RX queue mode'. This feature allows user-space applications to bypass the Linux kernel for specific network flows, enabling high-speed packet processing and filtering of large packet floods. This is crucial for their network infrastructure, especially when dealing with attacks, and addresses limitations with certain NICs by providing a more flexible kernel bypass solution.
  247. Kernel bypassThis post introduces the concept of 'kernel bypass' as a necessary technique to overcome the performance limitations of the vanilla Linux kernel networking stack for high-throughput packet processing. It details experiments demonstrating the kernel's limitations with multi-queue packet handling and then explores various kernel bypass techniques including PACKET_MMAP, PF_RING, Snabbswitch, DPDK, Netmap, Solarflare's EF_VI, bifurcated drivers, and virtualization approaches, highlighting their pros and cons in the context of Cloudflare's needs.
  248. Quick and dirty annotations for Go stack tracesThis post introduces a technique for improving the debuggability of Go stack traces by converting IPv4 addresses to their uint32 representation and passing this as a blank identifier parameter to goroutines. This allows engineers to quickly identify which network interface a specific goroutine is listening on by examining the stack trace, significantly aiding in troubleshooting issues within their large-scale Go applications like RRDNS.
  249. Introducing Partner AnalyticsThis post introduces the Partner Analytics portal, which provides hosting provider partners with granular data on bandwidth savings, threats blocked, and request/visitor metrics. This capability is enabled by an upgrade to Cloudflare's customer analytics and request logging backend, allowing for dynamic analytics data aggregation across all partner clients in rolling 7 and 30-day periods. The post details the breakdown of bandwidth (cached vs. uncached), threat types and origins, and request/visitor counts, highlighting the value Cloudflare delivers to its partners.
  250. Blue Light Special: Ensuring fast global configuration changesThis post introduces a novel, visual alerting system called the 'Blue Light Special' to monitor and ensure the rapid global propagation of configuration changes (like DNS records and WAF rules) across Cloudflare's network. It highlights the challenge of near-instantaneous updates and the use of a Raspberry Pi connected to a visible police light to provide a visceral, real-time indicator of propagation times, ensuring developers and operations teams are immediately aware of any delays.
  251. Setting Go variables from the outsideThis post introduces a more efficient method for embedding version information into Go binaries using the `-X` linker option, replacing the previous Makefile-based generation of `version.go`. This demonstrates an improvement in build tooling and a smarter approach to managing application metadata within the Go ecosystem.
  252. Go has a debugger—and it's awesome!This post introduces the `godebug` tool, a cross-platform debugger for Go programs. It highlights how Cloudflare uses `godebug` to debug its internal Go DNS server, RRDNS, demonstrating its effectiveness in improving development workflows and the ability to debug complex Go applications without requiring additional tools or permissions on the server. The post also touches upon the broader concept of source rewriting in Go tooling.
  253. CloudFlare "Interview Questions"This post contributes to the understanding of Cloudflare's deep engagement with low-level network protocols (TCP/IP stack, IPv4/IPv6, UDP, ICMP) by sharing interview questions. It reveals that Cloudflare engineers are expected to have a comprehensive understanding of these protocols, which is crucial for developing and maintaining their automatic attack mitigation systems. The post touches upon various aspects of network communication, including packet structure, flags, fragmentation, congestion control, and Linux-specific network configurations.
  254. Go crypto: bridging the performance gapThis post details Cloudflare's significant investment in optimizing Go's cryptography performance. It highlights the creation of assembly implementations for Elliptic Curves and AES-GCM to match or exceed OpenSSL's speed, addressing a critical need for high-volume encryption at Cloudflare's scale. The post also emphasizes the contribution of these performance improvements back to the Go community through a special fork, demonstrating a commitment to open source and shared advancement in cryptographic performance.
  255. Scaling out PostgreSQL for CloudFlare Analytics using CitusDBThis post details the scaling of Cloudflare's analytics database from a single PostgreSQL instance to a cluster using CitusDB. It outlines the log processing pipeline, including the use of Cap'n Proto, Go, Kafka, and CitusDB. The post explains the rationale behind choosing Go, Kafka, and PostgreSQL, and elaborates on the criteria for scaling PostgreSQL (performance, PostgreSQL compatibility, scaling, high availability, and business intelligence queries). It also describes the CitusDB architecture and the benefits of PostgreSQL extensions like HyperLogLog and Hstore on CitusDB.
  256. A Go Gotcha: When Closures and Goroutines CollideThis post highlights a common pitfall in Go programming when using closures with goroutines, specifically demonstrating how shared loop variables can lead to unexpected behavior. It provides a clear explanation and a practical solution by passing the loop variable as a function parameter, contributing to the understanding of robust concurrent programming practices within Cloudflare's data infrastructure.
  257. How Stacks are Handled in GoThis post details Cloudflare's use of Go and its approach to managing goroutine stacks. It explains the evolution from segmented stacks to stack copying, highlighting the performance implications and the ongoing efforts to rewrite the Go runtime in Go to enable features like concurrent garbage collection and improve stack copying capabilities. This contributes to the understanding of how Cloudflare optimizes its internal systems for performance and resource efficiency.
  258. Go interfaces make test stubbing easyThis post introduces the use of Go interfaces to simplify unit testing for components like image compression workers. By defining a `Job` interface, the `Worker` package can accept any type that implements `Do()` and `Priority()`, allowing for the creation of `DummyJob` implementations for testing purposes. This significantly improves test isolation and reduces the complexity of setting up test environments for components that interact with complex underlying systems.
  259. It's Go Time on LinuxThis post delves into the specifics of how Go programs on Linux obtain the current time, highlighting the evolution from older system calls like `time` and `gettimeofday` to the more precise `clock_gettime`. It details Cloudflare's adoption of the Linux vDSO (Virtual Dynamically linked Shared Objects) mechanism for low-overhead time retrieval, significantly improving performance for `time.Now()` and `runtime.nanotime`. The post also includes benchmarks demonstrating the performance gains and discusses the importance of accurate timekeeping for various system functions and potential pitfalls like Y2K and leap second bugs.
  260. Recycling memory buffers in GoThis post delves into memory management challenges in Go, a language heavily used at Cloudflare for network services. It details techniques for recycling memory buffers using channels and a dedicated recycler goroutine to reduce garbage collection overhead and improve memory efficiency, demonstrating a practical application of low-level optimization within Cloudflare's data infrastructure.
  261. A Tour Inside CloudFlare's Latest Generation ServersThis post details the evolution of Cloudflare's server hardware (G1-G4) and its components (CPU, storage, network) from 2010 to 2013. It highlights the specific design choices and optimizations made to handle the massive scale of traffic, including the adoption of SSDs, increased RAM for caching, low-power Intel Xeon CPUs, and high-performance Solarflare network cards with kernel bypass technology. This hardware advancement is crucial for enabling the data processing and analytics capabilities discussed in other posts.
  262. Integrating Kyoto Tycoon With PostgreSQLThis post details the technical challenge of propagating configuration rules across Cloudflare's global network of Points of Presence (PoPs) in near real-time. It introduces the use of PostgreSQL for storing these rules and Kyoto Tycoon (a distributed key-value store) for replication. The core contribution is the implementation of a writable Foreign Data Wrapper (FDW) for PostgreSQL 9.3, allowing direct integration and transactional operations with Kyoto Tycoon, thereby simplifying and accelerating the rule synchronization process.
  263. What CloudFlare LogsThis post details Cloudflare's initial logging policies and infrastructure. It explains that logs are generated for security threat identification and performance bottleneck analysis. For most customers, logs are discarded within 4 hours due to the massive scale (10 Petabytes/year and doubling every 4 months). Enterprise customers can opt to store logs for 3 days for export. Aggregate data is used for analytics (Cloudflare Analytics page) and security rule updates, with a stream processing engine correlating data at edge data centers. Error logs are kept for 1 week.
  264. Today's System-Wide UpgradeThis post details a significant system-wide upgrade focused on migrating Cloudflare's customer provisioning and accounting systems. The upgrade involved moving over half a million websites to a new, more modern system, rebalancing them across IP addresses, and seamlessly reissuing SSL certificates. This migration significantly enhanced the network's flexibility and robustness, enabling better isolation of sites under attack and facilitating compliance with regional content restrictions. It represents a major step in the operational maturity of Cloudflare's data infrastructure, allowing for more dynamic resource allocation and improved management of network resources.
  265. Pushing Nginx to its limit with LuaThis post introduces the use of Lua scripting within Nginx to augment its capabilities. It details how Lua can be embedded into Nginx to add custom functionalities, such as real-time log aggregation and processing, without resorting to custom C modules. This demonstrates an evolution in how Cloudflare processes and analyzes operational data at the edge, complementing its existing data infrastructure and analytics platforms by enabling more flexible and performant custom logic execution.
  266. Update: More Page View Counting RefinementThis post refines Cloudflare's page view counting algorithm. It addresses cases where objects were incorrectly classified as text/html when they were images or non-HTML, and fixes instances where 300 redirects were counted as page views, leading to double counting. The net impact is small for most sites but can be significant for others, bringing page view stats closer to beacon-based tracking systems while still reporting higher numbers due to crawler traffic.
  267. A Quick Update on Page ViewsThis post details a change to Cloudflare's logging system to exclude AJAX (XHR) calls from page view counts. Previously, these calls were treated as page views, leading to inflated numbers for sites with heavy AJAX usage. The update modifies how logs are counted at the network edge, resulting in a system-wide drop of approximately 17% in reported page views, with a more substantial decrease for AJAX-heavy sites to provide a more accurate representation.
  268. Understanding Analytics: When Is a Page View Not a Page View?This post clarifies Cloudflare's approach to analytics, differentiating its raw log-based hit and bandwidth tracking from beacon-based page view tracking (like Google Analytics). It explains the challenges of defining 'page views' in the context of modern AJAX-driven websites and highlights that Cloudflare's metrics provide a more accurate picture of server load and resource consumption, while acknowledging that beacon-based services are better suited for ad impression tracking.
  269. 2,000 Page Views Per SecondThis post marks a significant milestone in Cloudflare's operational scale, achieving 2,000 page views per second. It highlights the evolution of their data infrastructure from manual log monitoring to a sophisticated, real-time global traffic display, demonstrating an increased capacity for handling and visualizing massive amounts of network data. This reinforces the ongoing development of their data infrastructure and analytics capabilities to manage and understand network performance at scale.
  270. Using the CloudFlare API to pull visitor IPsThis post introduces the Cloudflare API's `zone_ips` function, allowing customers to programmatically retrieve visitor IP addresses from the past 48 hours. It details how to use the API with parameters for API key, zone ID, hours, and optional class (regular, crawler, threat) and geo-location data. The post also notes differences in data availability for Free vs. Pro accounts and suggests using the data for threat detection and blocking IPs via the Threat Control panel.
  271. CloudFlare, Now With Faster Stats!This post details an early iteration of Cloudflare's logging infrastructure, specifically addressing an I/O bottleneck in displaying statistics. To overcome slow query results, Fusion-IO cards were temporarily integrated into the core logging infrastructure to accelerate the parts of the stats database requiring high performance. This was a short-term fix to improve the speed of the stats page while longer-term storage architecture enhancements were planned.