TechPrufer
Get started
Cloudflare
Cloudflare logoCloudflare · Feature trail

Rate Limiting and Event Counting

13 posts · Aug 2011 – Oct 2026

What this trail covers

Cloudflare One's capabilities for identifying and controlling AI agent traffic (MCP) have evolved. This includes detecting MCP traffic based on protocol signals beyond simple hostnames/paths, distinguishing between approved MCP Portals and unmanaged connections, and providing granular controls at the client, network, and server levels. New features allow for the identification of MCP traffic even when it bypasses standard hostnames/paths, and enforce access policies based on protocol headers and

All Cloudflare trails
  1. Follow the thread: a new dashboard to investigate account abuseIntroduces a new dashboard for Account Abuse Protection (AAP) that provides a stateful trust model for fraud prevention. The dashboard offers an account population overview to identify suspicious trends and allows drilling down into individual account investigations. It visualizes login/signup volume, failed logins, leaked credential matches, and associated network/device signals. New roles (Account Abuse Protection and Account Abuse Protection PII) are introduced for granular access control.
  2. The Internet has a second audienceThis post details the significant shift in internet traffic composition, with AI agents now constituting over half of HTTP requests. It introduces Cloudflare's strategies and products to address this change: AI Crawl Control, Business Insights, BotBase, and Web Bot Auth for identifying and verifying AI agents; granular controls for search, agent, and training traffic; and the 'Disallow AI Training' mechanism. Crucially, it introduces two new monetization primitives: Pay Per Use, for direct publisher-AI deals and content licensing, and Monetization Gateway, for per-request pricing of APIs and services. These aim to create a 'payable' internet where content creators can monetize agent usage, moving beyond a binary 'yes' or 'no' to content access.
  3. Saving another 100TB of RAM with math (and Rust)This post details the optimization of consistent hashing in the Pingora Backend Router (PBR) service, specifically addressing excessive memory usage in the pingora-ketama library. The core problem identified was the memory footprint associated with representing servers with multiple hashes. The solution involved refining the consistent hashing algorithm by: 1. Increasing the number of hashes per server (from 1 to 160) to improve workload distribution and reduce the coefficient of variation. 2. Implementing server weights proportional to disk space (using ketama algorithm principles) to scale workload allocation based on resource availability. This optimization resulted in reclaiming over 100TB of RAM globally.
  4. Have it both ways: stay discoverable in search while disallowing AI trainingIntroduced the 'Disallow AI Training' setting within Bot Management, enabling granular control over AI training while preserving search discoverability. This setting leverages robots.txt and crawler classification (Search, Training, Agent) to differentiate crawler intent. The post details the 'Accountable' crawler designation, requiring mechanisms for opting out of AI training and summaries, URL-level visibility, and assurance that opting out does not impact search ranking. It outlines changes to existing settings ('Block', 'Block on pages with ads') and migration paths for existing customers, along with recommended configurations for new domains. Specific details on how Applebot, Googlebot, and Microsoftbot honor these new controls are provided.
  5. Introducing Adaptive Intelligence: undermining the economics of every bot attackIntroduces Adaptive Intelligence, a new bot detection engine that shifts from a 'tall wall' approach to making attacks costly and slow. It features continuous ML retraining on live traffic, disposable rule generation that injects noise into attacker feedback loops, and learning from customer corrections. The first component, a continuously retraining ML model, is launching today, measuring the probability of automated abuse by aggregating network signals (JA4 TLS fingerprints, request structures, challenge outcomes, session behavior, network reputation, client-side telemetry) over multiple time windows. Future components will include automated mining for disposable rule generation and learning from cross-site patterns.
  7. BotBase for Operators: A clearer path to joining Cloudflare's directory of bots and agentsThis post introduces BotBase for Operators, a new interface within the Cloudflare dashboard that provides bot operators with transparency and control over their bot submissions. Key technical contributions include: a dedicated UI for bot submissions, tracking submission status (Waiting for review, Accepted, Rejected) with reasons for rejection, the ability to edit existing submissions, and an automated review process that verifies IP lists, reverse DNS, and Web Bot Auth signatures. The submission form has been updated to capture bot behavior, content use (aligned with Content Signals), and operator classification (direct vs. intermediary).
  8. Say it once: introducing Bot Preference SyncIntroduces Bot Preference Sync, a feature that automatically updates a website's robots.txt file based on AI bot configuration settings (Search, Agent, Training) in the Cloudflare dashboard. This eliminates manual management of robots.txt for AI bot preferences, ensuring consistency between stated policies and edge enforcement. It also refines the 'Disallow Training' option to allow cooperating mixed-use crawlers to still access content for search indexing if they meet transparency requirements (respecting 'no training' in robots.txt, providing opt-outs for AI summaries, URL-level visibility into training data usage, and demonstrating that disallowing training doesn't harm search results). The feature defaults to 'on' for new customers and prompts existing customers using legacy managed robots.txt to transition.
  9. How Cloudflare detects MCP traffic and helps secure itThis post introduces new methods for detecting Model Context Protocol (MCP) traffic by analyzing protocol headers (`MCP-Protocol-Version`, `Mcp-Method`, `Mcp-Name`) and JSON-RPC message content, moving beyond simple hostname/path matching. It details how Cloudflare Gateway can identify MCP traffic, including 'shadow MCP traffic' that bypasses approved paths, and enforce policies based on these signals. The post also discusses control points within the MCP client, network gateway, and MCP server, highlighting the importance of server-side enforcement (e.g., WriteGuard) for comprehensive security. The evolution of MCP protocol specifications is noted as a factor in improving detection reliability.
  10. Unveiling good and bad behaviors on the Agentic InternetThis post details the Web Integrity & Trust team's strategy for assessing visitor trust based on continuous behavior analysis, moving beyond static checks. It introduces the Precursor system for detecting subtle inhuman bot traffic throughout a session, highlighting its ability to drive up the cost for bot developers. The post also announces Adaptive Intelligence, a new detection engine that will continuously learn and self-adjust based on traffic patterns. Additionally, it discusses advanced mitigations for influencing bot behavior, such as unpredictable responses and AI Labyrinth.
  11. Building an open Agentic Internet: readable, discoverable, callable, and payableThis post introduces the concept of the 'Agentic Internet' and outlines Cloudflare's role in building the necessary protocols and tools for agents to interact with the web. Key contributions include the development and promotion of open standards such as Web Bot Auth, PACT, WebMCP, x402, and Markdown for Agents. It also details the creation of Kitesurf, a browser for agents, and AI Search for discoverability. The post emphasizes the shift from human-centric web interactions to agent-centric ones, focusing on identity, discoverability, callability, and payability. It also discusses the evolution of bot detection and management, moving towards a framework where agents can be welcomed or blocked based on their behavior and adherence to new protocols.
  12. Give any website a WebMCP interfaceIntroduces WebMCP, a new browser standard that allows websites to expose tools for AI agents. Cloudflare provides an edge injection and bridge script to enable this for any website on Cloudflare, with no origin code changes. This enables agents to interact with websites differently than human users, and the initial release includes packs for Content Credentials (C2PA) and a Site MCP Server proxy. The bridge script uses HTMLRewriter to inject a script tag and serves a bridge.js worker at the edge. The bridge registers tools using `.registerTool` and can proxy calls to an origin MCP server.
  13. The next generation of MCPThe Model Context Protocol (MCP) has been updated to a stateless protocol, removing the need for stateful connections and simplifying server deployment to Cloudflare Workers. This change removes the requirement for Durable Objects for MCP servers and introduces Multi Round-Trip Requests (MRTR) for elicitations, replacing open streams. MCP requests now leverage HTTP headers (MCP-Protocol-Version, Mcp-Method, Mcp-Name) for gateway and rate limiter decisions, and include caching hints. Authorization has been tightened with preferred pre-registered clients, Client ID Metadata Documents (CIMD), and RFC 9207 issuer identification. A formal feature lifecycle (Active, Deprecated, Removed) has been introduced, with legacy transports and features like Dynamic Client Registration being deprecated.
  14. WriteGuard: Fine-grained controls for MCP ServersIntroduced WriteGuard, a shared policy, attribution, and auditing layer for AI agent interactions with MCP servers. WriteGuard allows for defining risk tiers (Read Only, Minimal Impact, Contained Write, Critical) for tools, enabling or disabling them, and configuring labeling for downstream applications. It enriches write actions with agent attribution and session context, preserving human identity while identifying the agent. WriteGuard asynchronously sends scrubbed audit events to an internal audit Worker, making agentic activity queryable across all MCP-enabled systems. The post details WriteGuard's application to GitLab tools like get_merge_request, create_mr_note, and merge_mr, demonstrating how it passes read-only calls, enriches contained writes, and blocks critical actions.
  15. Announcing Cloudflare Wallets: The programmable wallet for the agentic InternetIntroduces Cloudflare Wallets, a new programmable wallet system designed for AI agents. This system provides stable identifiers (handles) and native payment capabilities (via x402 protocol and stablecoins) for agents to interact with APIs and content. It includes Account Wallets for human users and Virtual Wallets for agents, with features for setting spending limits, allowances, and transaction caps. The system also introduces human-readable agent identifiers linked to Cloudflare accounts to improve merchant interaction and attribution.
  16. Introducing Precursor: detecting agentic behavior with continuous client-side signalsIntroduces Precursor, a client-side, session-based verification system that continuously collects behavioral signals (e.g., mouse movement, keyboard activity, focus changes, visibility) via injected JavaScript. These signals are processed in real-time on the edge to continuously distinguish human traffic from automated or agentic traffic. Precursor enhances bot protection by extending detection across the entire web application, complementing Turnstile and existing bot management features. It focuses on capturing consistent human behavior over time, which is difficult for bots to replicate, by analyzing patterns like wrist pivot, cognitive load delays, and hand tremor. The system includes an injection and collection layer, an edge-side evaluation layer with cross-referencing evaluators, and session integration that feeds into bot score and challenge decisions. Privacy is maintained by capturing minimal information and processing signals internally without exposing them to customer dashboards or tying them to user identities. New session-based analytics are introduced in Security Analytics to visualize full visitor journeys.
  17. Announcing the Monetization Gateway: charge for any resource behind Cloudflare via x402Introduces the Monetization Gateway, a new system for charging for any resource behind Cloudflare using the x402 protocol. It enables usage-based pricing, variable pricing, and charging unauthenticated callers at the edge. Payments settle in stablecoins, with settlement occurring peer-to-peer. The gateway handles payment verification and enforcement at the edge, reducing load on origins and simplifying billing for sellers.
  18. Content Independence Day, one year on- building the business model for the agentic InternetThis post analyzes the shift in internet traffic towards AI agents and the emergence of a market for monetized content. It details how AI adoption has accelerated, leading to a decline in human traffic to the open web and a significant increase in agent traffic, with over 50% of internet traffic now being non-human. The post highlights that 52% of crawler requests are now for AI training, up from 22% in Spring 2025. It discusses the breakdown of the traditional content monetization model, where referral traffic is declining as AI systems provide consolidated answers. Cloudflare's role in enabling transparency and control for site owners through attribution, business intelligence, and enforcement tools is emphasized, leading to scarcity and leverage for publishers in negotiating licensing agreements with AI companies. The post also points out inefficiencies in the current market, such as bespoke licensing and the challenge of content valuation, and critically examines Google's mixed-use crawler which combines discovery and AI access, limiting transparency for site owners.
  19. Your site, your rules: new AI traffic options for all customersThis post introduces a new taxonomy for classifying AI traffic based on bot behavior (Search, Agent, Training) and content use (immediate, reference, full). It also announces new default settings for blocking Training and Agent bots on pages with ads, and the launch of BotBase for Enterprise customers to provide visibility into known bots and their classifications. The core technical contribution is the classification system and the associated control mechanisms, rather than a new underlying system architecture.
  20. Unmasking the crawls with Attribution Business InsightsIntroduces the Attribution Business Insights dashboard, which provides website owners with granular data on bot traffic, including crawl-to-referral ratios per bot operator and classification of AI bots into Training, Search, or Agent categories. This enables informed decisions about content access and AI company relationships.
  21. Introducing the Cloudflare One stack- agent-powered deploymentIntroduced the Cloudflare One stack, a set of agent skills designed to automate the configuration, deployment, and management of Zero Trust environments. This stack includes skills for remote access and VPN replacement, user/network/device/data security, connectivity, migration guidance from legacy vendors (e.g., Zscaler), network diagram interpretation and generation, vendor concept translation, and troubleshooting/operations with the DEX toolkit. The stack leverages the Cloudflare code mode MCP server for a typed interface to the Cloudflare API, enabling agents to query and modify configurations through curated workflows.
  22. Scaling Security Insights: how we achieved a 10x increase in global scanning capacityThis post details the architectural and system design changes made to scale Cloudflare's Security Insights platform to achieve a 10x increase in global scanning capacity. Key contributions include: implementing batch processing and parallel goroutines in Kafka checkers to avoid head-of-line blocking, introducing a 'slow lane'/'fast lane' system to handle variable processing times, optimizing Postgres inserts using a hybrid UNNEST/COPY approach, resolving API latency issues by switching to an active-passive setup aligned with the primary database, and rearchitecting the scheduler to independently schedule zones, randomize scan times, and implement adaptive rate limiting for uniform scan distribution.
  23. Moving past bots vs. humansThis post discusses the evolving landscape of web traffic, moving beyond the 'bots vs. humans' dichotomy to focus on 'intent and behavior'. It highlights how AI agents bypass traditional browser rendering, making traffic opaque to website owners. The post explains the client-server model and the inherent uncertainty it creates regarding client intent. It details current bot management strategies based on passive and active client signals, and server observations, emphasizing the limitations of IP-based access control and the transformation of mitigation vectors into tracking vectors. The post introduces the 'rate limit trilemma' (decentralized, anonymous, accountable) and argues that the current web model leans towards decentralized and anonymous, necessitating more sophisticated intent-based controls.
  24. Redirects for AI Training enforces canonical contentIntroduces 'Redirects for AI Training', a feature that leverages canonical tags and verified AI crawler identification to issue HTTP 301 redirects to up-to-date content. This feature aims to improve the quality of AI training data by ensuring crawlers ingest current documentation. It also enhances Radar's AI Insights page with Response Status Code analysis for AI crawlers.
  25. Scaling MCP adoption: Our reference architecture for simpler, safer and cheaper enterprise deployments of MCPThis post introduces a reference architecture for securing enterprise MCP deployments. Key contributions include: 1. The concept of 'remote MCP servers' deployed on Cloudflare's developer platform for centralized management, visibility, and low-latency access. 2. The integration of Cloudflare Access for authentication and SSO/MFA for private MCP servers. 3. The introduction of 'MCP server portals' for centralized discovery, governance, logging, and DLP, supporting both remote and third-party MCP servers. 4. The 'Code Mode' pattern for MCP server portals, collapsing thousands of API endpoints into two tools (`portal_codemode_search` and `portal_codemode_execute`) to drastically reduce token costs and context window usage. 5. The use of Cloudflare Gateway for Shadow MCP detection to identify unauthorized remote MCP server usage. The architecture emphasizes running security and networking components on the same physical machine for performance.
  26. From bytecode to bytes- automated magic packet generationThis post introduces a novel approach to reverse-engineering classic BPF filters used in Linux malware by employing symbolic execution with the Z3 theorem prover and the scapy Python library. It details the process of analyzing BPF instructions, identifying paths to acceptance conditions, and automatically generating the specific network packets required to trigger these filters. The authors present a Python class `BPFPacketCrafter` that implements the symbolic execution machine and demonstrates its application with the BPFDoor malware sample, translating Z3 solver constraints into scapy-crafted packets.
  27. Slashing agent token costs by 98% with RFC 9457-compliant error responsesIntroduced RFC 9457-compliant structured Markdown and JSON error responses for all 1xxx-class Cloudflare platform errors. This provides machine-readable instructions for AI agents, including error classification, retryability, and escalation guidance, replacing verbose HTML error pages. The new format reduces payload size and token usage by over 98% compared to HTML, with stable fields for deterministic agent control flow.
  28. AI Security for Apps is now generally availableIntroduced AI Security for Apps, a product that detects and mitigates threats to AI-powered applications. Key features include: automatic discovery of AI endpoints (now free for all plans), detection of prompt injection, PII exposure, and toxic topics. New GA capabilities include custom topic detection and custom prompt extraction. Mitigation is handled via WAF rules, combining AI-specific signals with other request data. Expanded partnerships with IBM and Wiz.
  29. Fixing request smuggling vulnerabilities in Pingora OSS deploymentsThis post details the patching of three HTTP/1.x request smuggling vulnerabilities (CVE-2026-2833, CVE-2026-2835, CVE-2026-2836) in the Pingora open-source framework. The vulnerabilities allowed attackers to bypass proxy-layer security controls, desync HTTP requests/responses, and poison caches in standalone Pingora deployments. The fixes include: 1. Ensuring subsequent bytes on an HTTP connection are only interpreted as an upgraded stream after a '101 Switching Protocols' response is received. 2. Strictly adhering to request length guidelines, ensuring HTTP request bodies are never considered close-delimited, and rejecting invalid HTTP/1.0 + Transfer-Encoding request messages. 3. Rejecting CONNECT requests by default due to their special message framing rules. The post also clarifies that Cloudflare's CDN was not affected due to its architecture and internal Pingora service configurations.
  30. The most-seen UI on the Internet? Redesigning Turnstile and Challenge PagesThis post details the redesign of Cloudflare's Turnstile widget and Challenge Pages, focusing on improving the user experience for billions of daily interactions. Key contributions include: establishing a unified information architecture across all states and pages, conducting user research to validate design decisions (e.g., favoring 'Verify you are human' over 'I am human'), and identifying critical areas for improvement such as replacing 'Send Feedback' with 'Troubleshoot' to offer actionable help. The redesign aims to reduce cognitive load and frustration for users during security verification processes.
  31. Innovating to address streaming abuse — and our latest transparency reportThis post details Cloudflare's technical innovations in addressing streaming abuse and copyright violations. It describes the development of an API for streamlined reporting from rightsholders, feedback mechanisms to improve report quality, and real-time response capabilities. The post highlights the use of machine learning and AI to identify abuse patterns and protect customers, including the expansion of automated systems to respond to technical abuse like phishing. It also details how information from streaming abuse reports is used to bolster technical tools for preventing unauthorized streaming over Cloudflare's network by websites using non-hosted services. The post emphasizes the importance of partnerships and information sharing with rightsholders to identify and respond to unauthorized streaming, and how this information is also useful in broader cybersecurity efforts.
  32. Cloudflare outage on December 5, 2025This post details a critical incident where a change to the WAF's body parsing logic and the disabling of an internal testing tool via a global configuration system led to widespread HTTP 500 errors. The root cause was identified as a bug in the FL1 proxy's rules module where a killswitch applied to a rule with an 'execute' action resulted in an attempt to index a nil value in Lua. The incident highlights the failure of the global configuration system's lack of gradual rollouts and the need for enhanced rollout and versioning, streamlined break-glass capabilities, and 'fail-open' error handling.
  33. Go and enhance your calm- demolishing an HTTP:2 interop problemThis post details a specific misconfiguration in Go's standard library HTTP client that leads to unnecessary RST_STREAM frames followed by PING frames, triggering Cloudflare's ENHANCE_YOUR_CALM mitigation. The root cause identified is calling `resp.Body.Close()` without fully reading the response body, especially when the body is empty. A fix involving `io.Copy(io.Discard, resp.Body)` before closing is provided. The post also explains why internal microservices communicate over the Cloudflare edge for dogfooding, authentication via Cloudflare Access, and integration with Workers.
  34. Anonymous credentials- rate-limiting bots and agents without compromising privacyThis post introduces the concept of anonymous credentials (AC) as a privacy-preserving mechanism for rate-limiting and managing AI agent traffic. It describes how ACs, currently under development at the IETF, can enable website operators to enforce security policies like rate-limiting or blocking specific malicious users without identifying or tracking individual users across requests. The post also provides code examples of a simple AI agent using Cloudflare Workers and Browser Rendering, and discusses the challenges origins face with the increased speed and potential maliciousness of AI agent traffic.
  35. Policy, privacy and post-quantum: anonymous credentials for everyoneThis post explores the application of post-quantum (PQ) cryptography to Anonymous Credentials (ACs) for private rate-limiting of AI agent traffic. It analyzes the requirements for ACs, including unlinkability, unforgeability, and state management (counters, nullifiers, public attributes), and evaluates the readiness of current PQ research for these demands. The post contrasts classical AC approaches with the challenges posed by PQ algorithms, highlighting the need for new cryptographic primitives and standardization efforts to bridge the gap between theory and real-world, high-scale deployments.
  36. Securing agentic commerce: helping AI Agents transact with Visa and MastercardThis post introduces the integration of Cloudflare's Web Bot Auth and HTTP Message Signatures with Visa's Trusted Agent Protocol and Mastercard's Agent Pay to secure agentic commerce. It details how AI agents can be cryptographically authenticated using public key cryptography and HTTP Message Signatures, with registration and key management handled by payment networks. The post outlines the validation checks performed by Cloudflare (signature presence, key retrieval, timestamp validation, nonce uniqueness, tag validation, signature verification) and provides pseudocode for a Cloudflare Worker demonstrating the signing logic. It also announces upcoming managed rulesets for merchants and integration with the Agent SDK.
  37. A deep dive into BPF LPM trie performance and optimizationThis post details performance bottlenecks in the Linux kernel's BPF LPM trie implementation, specifically slow lookup, update, delete, and free operations, which have caused production issues like soft lockups and packet loss. It explains the trie data structure, its use in IP routing and Cloudflare's Magic Firewall, and the limitations of the current BPF LPM trie implementation, such as its 2-child node design, lack of level compression, and resulting performance degradation with increasing entry counts. Benchmark results and analysis of lookup throughput degradation are provided.
  38. Giving users choice with Cloudflare’s new Content Signals PolicyIntroduces the Content Signals Policy, an extension to robots.txt that allows website operators to define machine-readable signals for how their content can be used by crawlers. Defines three signals: 'search', 'ai-input', and 'ai-train', with 'yes'/'no' values. This policy aims to address the free-rider problem and the dilemma of locking down content versus allowing open access, particularly in the context of AI model training and data scraping. Cloudflare will automatically update robots.txt for managed robots.txt customers to disallow AI training and will serve the human-readable policy for free customers without existing robots.txt files.
  39. To build a better Internet in the age of AI, we need responsible AI bot principles. Here’s our proposal.This post proposes five responsible AI bot principles: public disclosure, self-identification (with a focus on cryptographic verification via Web Bot Auth), declared single purpose, respecting preferences, and acting with good intent. It highlights the impact of AI crawlers on web publishers and proposes a framework for AI bot behavior to ensure transparency, accountability, and respect for content access and use preferences. It also references existing Cloudflare initiatives like Content Independence Day, pay-per-crawl, and verified bots policy.
  40. Building unique, per-customer defenses against advanced bot threats in the AI eraThis post introduces a new platform for per-customer behavioral anomaly detection to combat AI-driven bot threats. It details a three-step process: establishing a dynamic baseline of normal traffic for each customer, identifying deviations from this baseline (e.g., systematic scraping of product pages, rapid article URL hits), and generating actionable findings via new Bot Detection IDs for WAF rules. It also highlights improvements in detecting residential proxy networks by combining network data with client-side fingerprinting from challenge solves, and mentions the use of heuristics like HTTP/2 fingerprints and Client Hello extensions for global bot fingerprinting.
  41. Helping protect journalists and local news from AI crawlers with Project GalileoProject Galileo participants, including news organizations, are now provided with free access to Cloudflare's Bot Management and AI Crawl Control services. This enables them to monitor and control AI crawler access to their websites, analyze crawler activity by type (e.g., AI search, AI scraper) and origin (e.g., Google, OpenAI, Anthropic), and make informed decisions about data access and potential monetization. This extends the existing capabilities of Bot Management and AI Crawl Control by integrating them into a program focused on protecting civic voices.
  42. Launching the x402 Foundation with Coinbase, and support for x402 transactionsThis post introduces the x402 protocol, a new framework for machine-to-machine payments over HTTP, and details its integration with Cloudflare's Agents SDK and MCP servers. It proposes a deferred payment scheme for x402, allowing for flexible settlement of agentic payments. The post also provides code examples for implementing x402 clients and servers, and highlights the x402 playground demo.
  43. The age of agents: cryptographically recognizing agent trafficIntroduces 'signed agents' as a new classification for automated traffic, distinct from existing verified bots. This classification leverages Web Bot Auth for cryptographic verification of HTTP requests, allowing origins to identify agents directed by end-users. The post details the application process via the Cloudflare dashboard and highlights partnerships with ChatGPT agent, Goose, Browserbase, and Anchor Browser. It also announces upcoming firewall rule integration for granular control over signed agents as a group and updates the Radar Bots Directory to include this new classification.
  44. The next step for content creators in working with AI bots: Introducing AI Crawl ControlThis post announces the general availability of AI Crawl Control, rebranding the AI Audit tool. The core technical contribution is the integration and customization of HTTP 402 'Payment Required' status codes for AI crawlers. This allows content creators to send specific messages to crawlers indicating how to license content, moving beyond simple blocking. The feature is configurable within the AI Crawl Control dashboard for paid Cloudflare customers, enabling them to select bots and customize the 402 response message.
  45. Securing the AI Revolution: Introducing Cloudflare MCP Server PortalsIntroduces Cloudflare MCP Server Portals, a new capability for centralizing, securing, and observing all MCP connections within an organization. This feature acts as a single gateway for MCP servers, integrating with Cloudflare One for policy enforcement (MFA, device posture, geo-restriction), providing aggregated logs for visibility and auditing, enabling a curated AI user experience based on least privilege through administrator approval of MCP servers, and simplifying user configuration by providing a single Portal endpoint. It addresses risks like prompt injection, supply chain attacks, privilege escalation, and data leakage associated with decentralized MCP implementations. The post also outlines future plans for enhanced AI security, including MCP server lockdown mechanisms, WAF integration for prompt injection detection, Cloudflare-hosted MCP servers with AI Gateway integration, ML-based abuse detection, and MCP protocol enhancements.
  46. Best Practices for Securing Generative AI with SASEThis post details how Cloudflare's SASE platform can be extended to govern the use of Generative AI, focusing on three pillars: Visibility, Risk Management, and Data Protection. It introduces new AI Security Posture Management (AI-SPM) features including shadow AI reporting, confidence scoring of AI providers, AI prompt protection, API CASB integrations, and tools to secure Model Context Protocol (MCP) deployments. The post outlines how to develop an AI Security Strategy by considering risk tolerance, sensitive data, compliance frameworks, and the use of autonomous AI agents. It emphasizes the foundational role of SASE in discovering and inventorying AI tools, monitoring prompts and responses, and enforcing Data Loss Prevention (DLP) policies to prevent data leakage. The Secure Web Gateway (SWG) is highlighted for redirecting traffic from unsanctioned AI providers to sanctioned enterprise AI providers. The integration of MCP tooling into the SASE platform is also discussed for securing agentic AI deployments.
  47. Unmasking the Unseen: Your Guide to Taming Shadow AI with Cloudflare OneThis post introduces the Cloudflare Shadow AI Report, a new feature within Cloudflare One that provides visibility into the usage of AI services. It details the technical implementation of this feature, including the use of Cloudflare Gateway for traffic inspection and classification of AI services, and the adoption of TimescaleDB for storing and querying large volumes of application usage data. The post also describes the development of materialized views for Shadow IT and AI use cases, and the integration with Cloudflare Gateway policies for enforcement based on application approval statuses (Approved, Unapproved, In Review). It highlights the use of DLP and Browser Isolation for managing 'In Review' applications and the availability of raw log data in Log Explorer for forensic investigations.
  48. Reducing double spend latency from 40 ms to < 1 ms on privacy proxyThis post details the investigation and resolution of high latency (~40 ms) in double-spend checks for Privacy Pass tokens used in Cloudflare's privacy proxy product. The root cause was identified as the interaction between Nagle's algorithm and delayed ACKs in TCP, where multiple small writes to `memcached` commands were being buffered and delayed. The fix involved using `BufWriter<TcpStream>` to buffer writes in user-space and send the entire command in a single `flush` operation, reducing latency to <1 ms.
  49. Perplexity is using stealth, undeclared crawlers to evade website no-crawl directivesThis post details how Perplexity is evading website no-crawl directives by using stealth crawlers with obfuscated user agents (generic browser impersonating Chrome on macOS) and rotating source IP addresses and ASNs. It describes Cloudflare's testing methodology using new domains with strict robots.txt and WAF rules, and how Perplexity's undeclared crawlers still accessed content. Cloudflare has de-listed Perplexity as a verified bot and added heuristics to block this stealth crawling behavior. The post also contrasts this with OpenAI's adherence to best practices for bot operators.
  50. Control content use for AI training with Cloudflare’s managed robots.txt and blocking for monetized contentThis post introduces two new tools for controlling AI bot access: a managed robots.txt file generation service and a feature to block AI bots from monetized content sections. It details the imbalance in crawl-to-referral ratios between AI training crawlers and traditional search crawlers, highlighting the need for publishers to protect their content. The post also presents updated data on AI bot traffic and usage of Cloudflare's existing AI bot blocking features, and discusses the technical aspects of robots.txt directives (User-agent, Disallow, Allow, Sitemap) and RFC 9309.
  51. From Googlebot to GPTBot: Who’s crawling your site in 2025This post analyzes the growth and trends of AI and search crawlers, specifically highlighting the surge of GPTBot and Meta-ExternalAgent. It provides data on their market share shifts and overall traffic growth, emphasizing the increasing relevance of AI-driven crawling. The post also touches upon the voluntary nature of robots.txt adherence and Cloudflare's tools like AI Audit for enforcement, which directly relates to managing and controlling AI agent traffic.
  52. Introducing pay per crawl: Enabling content owners to charge AI crawlers for accessIntroduced 'pay per crawl' feature, leveraging HTTP response code 402 to enable content owners to charge AI crawlers for access. Implemented publisher controls for pricing and access rules (Allow, Charge, Block). Integrated with Web Bot Auth for secure crawler identification via Ed25519 keys and HTTP Message Signatures. Defined reactive (discovery-first) and proactive (intent-first) payment flows using `crawler-price`, `crawler-exact-price`, and `crawler-max-price` headers. Cloudflare acts as Merchant of Record for financial settlement.
  53. Message Signatures are now part of our Verified Bots Program, simplifying bot authenticationThis post announces the integration of HTTP Message Signatures into Cloudflare's Verified Bots Program. It details how bots can use Message Signatures to cryptographically sign requests for faster enrollment and verification. For site owners, Cloudflare will automatically validate these signatures at the edge, marking verified bots for use in Bot Management and WAF rules. The post also introduces open-source libraries (Rust and TypeScript) for generating Message Signatures and outlines the steps for bots to register and for Cloudflare to verify these signatures, including the use of key directories and the `Signature-Agent`, `Signature-Input`, and `Signature` headers.
  54. Resolving a request smuggling vulnerability in PingoraThis post details a request smuggling vulnerability (CVE-2025-4366) discovered in the Pingora OSS framework, specifically within the pingora-cache crate. The vulnerability arose from a bug in HTTP/1.1 parsing when caching was enabled, causing unread request bodies on cache hits to be misinterpreted as subsequent requests. This could lead to request header modification and URL hijacking. Cloudflare mitigated the vulnerability by disabling traffic to vulnerable components, patching Pingora to version 0.5.0, and invalidating cached assets. The post also explains request smuggling and its impact on CDN services, including a secondary attack effect where origin servers could redirect users to malicious sites, revealing visited URLs.
  55. Forget IPs: using cryptography to verify bot and agent trafficIntroduces HTTP Message Signatures (RFC 9421) and request mTLS as cryptographic mechanisms for authenticating AI agent and bot traffic. Details the implementation of HTTP Message Signatures, including the `Signature-Input` and `Signature-Agent` headers, validity windows, Key IDs, and tags. Provides examples of generating signatures using `chrome.webRequest.onBeforeSendHeaders` and a helper npm package `web-bot-auth`. Discusses the trade-offs of signature overhead versus existing bot mitigation costs. Proposes a new tag `web-bot-auth` for this specific use case.
  56. Build and deploy Remote Model Context Protocol (MCP) servers to CloudflareThis post introduces the ability to build and deploy remote Model Context Protocol (MCP) servers on Cloudflare Workers. It details four key components that facilitate this: workers-oauth-provider for OAuth 2.1 authorization, McpAgent for remote transport handling, mcp-remote adapter for local client compatibility, and an AI playground as a remote MCP client. The post explains the MCP OAuth flow, the role of workers-oauth-provider in abstracting OAuth implementation, and the rationale behind MCP servers issuing their own tokens to enforce granular access controls and mitigate 'Excessive Agency' risks. It provides code examples for integrating GitHub as an identity provider and restricting tool access based on user allowlists.
  57. Simplify allowlist management and lock down origin access with Cloudflare AegisThis post introduces Bring Your Own IPs (BYOIP) for Cloudflare Aegis, a customer-accessible API, and observability for Aegis IP utilization. It details how Aegis provides dedicated egress IP addresses for origin protection, allowing customers to lock down their origin firewalls by allowlisting a small set of IPs. The post explains the routing of requests with Aegis, including the use of soft-unicasting and the L4 XDP-based load balancer, Unimog. It addresses capacity constraints, estimating ~40k concurrent connections per Aegis IP address per origin to prevent port exhaustion. The post also elaborates on Aegis with IPv6, where customers receive two /64 prefixes for global allocation, and Aegis with IPv4, where customers work with Cloudflare to determine the ideal number of IP addresses, with BYOIP now supporting entire IPv4 /24 prefixes or IPv6 /64 blocks.
  58. Upgraded Turnstile Analytics enable deeper insights, faster investigations, and improved securityThis post introduces a major upgrade to Turnstile Analytics, providing enhanced visibility into bot traffic. New features include TopN statistics for various traffic attributes (hostname, ASN, user agent, browser, source IP, country, OS), detailed challenge outcomes (likely human vs. likely bot), solve rates (interactive, non-interactive, pre-clearance), and token validation results. These upgrades aim to enable faster identification of bots, finer tuning of security postures, and proactive mitigation strategies by correlating Turnstile data with other security measures like WAF rules. The post also provides a real-world example of setting up Turnstile for login/signup pages and interpreting the new analytics to monitor for account takeover and credential stuffing attacks.
  59. Enhanced security and simplified controls with automated botnet protection, cipher suite selection, and URL Scanner updatesThe Security Level feature has been automated by integrating botnet data and request rate signals, removing the need for manual threshold configuration. This enhancement leverages a new security system that combines real-time DDoS score, traffic threshold, and a botnet tracking system, which has been deployed across all customer traffic. Cipher suite selection is now available in the Cloudflare dashboard, offering three user flows: security recommendations (Modern, Compatible), compliance presets, and custom selections. The URL scanner has been updated with new features including bulk scanning, similarity search, and a location picker.
  60. Banish bots from your Waiting Room and improve wait times for real usersThis post introduces two new capabilities for Cloudflare Waiting Room: integration with Turnstile for bot detection and the Infinite Queue feature for Advanced Waiting Rooms. Turnstile is integrated into the queuing page to challenge traffic and gather analytics on bot activity. Traffic failing the Turnstile challenge can be sent to an Infinite Queue page, which mimics a real user experience to waste bot resources without impacting real users. Session Revocation is also introduced, allowing users to end a session based on an action (e.g., purchase completion) to dynamically open up spots in the queue. This feature improves throughput, decreases wait times, and increases fairness by allowing more users to pass through the queue during high demand events. New metrics for bot traffic and challenge outcomes are available in Waiting Room Analytics and the GraphQL API.
  61. Grinch Bots strike again: defending your holidays from cyber threatsThis post details the increased prevalence and sophistication of bots, including AI bots, during holiday shopping periods (Black Friday, Cyber Monday). It quantifies bot traffic as a percentage of total traffic and login attempts, and highlights the use of residential proxy networks and AI bots for scraping, inventory hoarding, and account takeovers. It also provides specific recommendations for customers to protect themselves, including blocking bots, monitoring for account takeovers, rate-limiting critical paths, and blocking AI bots. The post also mentions the evolution of Cloudflare's Bot Management and account abuse detection capabilities in response to these threats.
  62. Robotcop: enforcing your robots.txt policies and stopping bots before they reach your websiteThis post introduces the "Robotcop" feature, which extends AI Crawl Control to enforce robots.txt policies for AI bots at the network level. It parses a site's robots.txt file, matches its rules against observed AI bot traffic, and translates non-compliant rules into WAF rules for enforcement. This moves beyond voluntary compliance to programmatic enforcement of content access policies for AI crawlers.
  63. AI Everywhere with the WAF Rule Builder Assistant, Cloudflare Radar AI Insights, and updated AI bot protectionIntroduced an AI Assistant for the WAF Rule Builder that generates custom and rate-limiting rules from natural language prompts, currently in beta. Added AI bot and crawler traffic insights to Cloudflare Radar, tracking top known AI bots based on the ai.robots.txt list. Enhanced AI bot blocking capabilities by expanding the rule to include 27 new signatures of AI bots that do not follow rules, converting this logic to a Cloudflare managed rule for real-time updates. Improved ML model detection rates for AI bots by incorporating additional heuristics alongside machine learning, achieving 100% correct scoring for requests. Observed a reduction in AI bot traffic, with an average decrease of 30 million requests daily and a decline in specific bots like Bytespider, attributed to Cloudflare's blocking features. Announced upcoming Terraform support for AI bot protection configuration.
  64. TURN and anycast: making peer connections work globallyThis post announces the general availability of Cloudflare Calls TURN service, which leverages Cloudflare's anycast network for global coverage and near-zero latency. It details how TURN addresses NAT unpredictability and privacy concerns by relaying traffic through Cloudflare's network, masking client IP addresses. The service auto-scales and is inherently global. For stateful protocols like TURN, a Unimog load balancer ensures consistent routing within a data center, and a middle-ground solution forwards misrouted packets over the backbone to the correct datacenter.
  65. Introducing Ephemeral IDs: a new tool for fraud detectionIntroduced Ephemeral IDs, a new feature that generates short-lived, client-specific identifiers independent of network-level information (like IP addresses). This enhances fraud detection by allowing the grouping of fraudulent actions from the same client, improving detection rates and reducing false positives, even when attackers rotate IP addresses. Ephemeral IDs are integrated into Turnstile and are available via the siteverify response for advanced use cases like preventing fraudulent account signups. This feature complements existing bot detection mechanisms by providing an additional signal beyond IP addresses and browser attributes.
  66. Start auditing and controlling the AI models accessing your contentThis post introduces new tools for site owners to audit and control how AI models access their content. It provides analytics on AI bot activity (by provider, bot type, and accessed content), a one-click option to block all AI bots, granular controls to allow specific AI providers or bot types via WAF rules, reporting for auditing AI scanning activity, and previews monetization features for content usage by AI models. The post details how AI bots identify themselves (User-Agent headers, IP addresses, behavior) and categorizes them (AI Data Scraper, AI Search Crawler, Archiver).
  67. Protecting APIs from abuse using sequence learning and variable order Markov chainsThis post details the application of sequence learning techniques, specifically variable order Markov chains (VOMCs), to detect sequential abuse in API requests. It explains how sessions are modeled, the limitations of standard Markov chains, and how VOMCs address these by adapting model order based on context. The post introduces the concept of credible intervals for estimating probabilities and highlights the use of VOMCs to identify statistically significant sequences that might be missed by volume-based analysis, enabling the suggestion of mitigation rules for API Shield's Sequence Mitigation product.
  68. A good day to trie-hard: saving compute 1% at a timeIntroduced a new open-source Rust crate, 'trie-hard', which implements a custom trie data structure optimized for prefix searches. This trie reduces the average runtime for clearing internal HTTP headers from 3.65µs to 0.93µs, resulting in a 1.28% reduction in overall CPU utilization for the pingora-origin service. The implementation stores node relationships in bits of unsigned integers and keeps the tree in contiguous memory for performance gains. Benchmarking and production stack trace sampling confirmed the performance improvements.
  69. Bringing insights into TCP resets and timeouts to Cloudflare RadarThis post introduces a new dashboard and API endpoint on Cloudflare Radar that provides a near real-time view of TCP connections to Cloudflare's network that terminate within the first 10 ingress packets due to resets or timeouts. It details the classification of these 'anomalous' TCP connections based on their stage (Post-SYN, Post-ACK, Post-PSH, Later) and discusses potential causes such as scanners, application shutdowns, network errors, attacks (e.g., SYN flood), and connection tampering by middleboxes. The post also provides guidance on how to use and interpret this dataset for self-evaluation, exploring new targets, and longitudinal studies.
  70. Go wild: Wildcard support in Rules and a new open-source wildcard crateIntroduced wildcard support ('wildcard' and 'strict wildcard' operators) to the Ruleset Engine, enabling flexible pattern matching across various string fields. Developed the `wildcard_replace()` function for Single Redirects to dynamically construct target URLs using captured wildcard segments. Implemented a custom Rust crate for wildcard matching, based on iterative algorithms, to support byte-oriented matching, UTF-8 strings, escape sequences, and capture segments, while managing performance complexity with a limit on asterisk metacharacters.
  71. Advancing Threat Intelligence: JA4 fingerprints and inter-request signalsThis post introduces JA4, a successor to JA3, for TLS fingerprinting. It details the limitations of JA3, including TLS extension randomization and limited scope, and explains how JA4 addresses these by incorporating ALPN and being protocol-agnostic. The post also describes Cloudflare's internal development of a new Rust-based parser, `client-hello-parser`, to efficiently and securely parse TLS ClientHello messages, enabling the integration of JA4 fingerprints across Cloudflare's platform. The parser's optimizations include amortized zero heap allocations, memory safety, and ultra-low latency through SIMD instructions and efficient iteration.
  72. RADIUS/UDP vulnerable to improved MD5 collision attackThis post details an improved MD5 collision attack that exploits RADIUS/UDP authentication modes (PAP, CHAP, MS-CHAP), allowing a Man-in-the-Middle attacker to gain unauthorized administrative access to network devices. The attack leverages the ad hoc MD5 construction of the RADIUS Response Authenticator, which is vulnerable to length extension attacks and collisions. The post explains the attack mechanism, its implications for network security, and provides mitigation strategies such as migrating to RADIUS/TLS or DLTS, and implementing stricter access controls for existing RADIUS/UDP deployments.
  73. Declare your AIndependence: block AI bots, scrapers and crawlers with a single clickThis post introduces a new one-click feature to block all AI bots, including those that impersonate legitimate visitors. It leverages Cloudflare's global machine learning models to identify bot activity regardless of user agent spoofing. The post details the prevalence of AI bot activity, identifies top AI crawlers like Bytespider and GPTBot, and analyzes website operator responses to AI bot traffic. It also explains how Cloudflare's bot detection models score traffic, even when user agents are spoofed, and provides mechanisms for customers to report misbehaving AI crawlers.
  74. Cloudflare incident on June 20, 2024This post details a critical incident where a new DDoS mitigation rule, designed to use rate limits and cookies for legitimate clients, exposed a latent bug in the rate-limiting component. The bug, caused by an unanticipated internal API usage and a discrepancy in cookie validation logic, led to infinite loops in HTTP request handling processes due to proper tail calls in Lua. This resulted in poisoned processes consuming 100% CPU and unable to process further requests, leading to increased latency and error rates across Cloudflare's network. The incident response involved disabling the problematic DDoS rule, restarting affected services, and leveraging Traffic Manager for traffic re-routing. The post also provides technical details on the rate-limiting key generation and the interaction between the DDoS rule workflow and the rate-limiting component.
  75. Using machine learning to detect bot attacks that leverage residential proxiesThe Bot Management machine learning model (v8) has been updated to detect bot attacks leveraging residential proxies. This involves analyzing network-level discrepancies (e.g., latency) and behavioral signals to identify traffic patterns indicative of residential proxy abuse. The model training process utilizes Apache Airflow for orchestration, Catboost for model training, and specializations for validation datasets to ensure accuracy across different traffic types. Features used include global aggregates, high cardinality features, and single request features.
  76. Disrupting FlyingYeti's campaign targeting UkraineThis post details Cloudforce One's proactive disruption of the FlyingYeti threat actor's phishing campaign targeting Ukraine. It describes the campaign's use of debt-themed lures, the exploitation of the WinRAR vulnerability (CVE-2023-38831), and the deployment of PowerShell malware (COOKBOX). Cloudforce One's countermeasures involved detecting and taking down malicious Cloudflare Workers, collaborating with third parties to remove cloud-hosted malware, and analyzing the actor's reconnaissance and weaponization activities. The post highlights how these actions prolonged the actor's operational timeline and prevented the successful delivery of their payload. It also provides technical details on the phishing site infrastructure, the malicious RAR file's contents, and the decoy documents used.
  77. New tools for production safety — Gradual deployments, Source maps, Rate Limiting, and new SDKsIntroduced a new API for direct access to rate limits from within Cloudflare Workers, enabling runtime control over rate limiting behavior. This API is backed by memcached and allows for defining rate limits programmatically within Worker code.
  78. New! Rate Limiting analytics and throttlingThis post introduces two key enhancements to Cloudflare's Rate Limiting capabilities: Rate Limit Analysis within Security Analytics and the new 'throttle' behavior. Rate Limit Analysis provides a dedicated tab to help users determine appropriate rate thresholds by visualizing request rate distributions based on IP, JA3, or both, over various time intervals. The 'throttle' behavior offers an alternative to the fixed action, selectively dropping requests to keep client rates below a defined threshold, mitigating issues with shared IPs like NAT. The post details how to use the new analytics to define rules and explains the difference between fixed action and throttle behaviors.
  79. Argo Smart Routing for UDP: speeding up gaming, real-time communications and moreThis post introduces the extension of Argo Smart Routing to accelerate UDP traffic. Previously focused on HTTP and TCP, Argo's route computation and congestion avoidance algorithms have been adapted for UDP. The post details performance improvements observed in testing, showing up to 17.3% reduction in round-trip-time for UDP traffic between Tokyo and Iowa, effectively reducing latency by over a thousand miles. This is achieved by applying real-time global network performance measurements and congestion data to UDP packet routing.
  80. Understand the impact of Waiting Room settings with Waiting Room AnalyticsThis post introduces Waiting Room Analytics, a new feature that provides insights into end-user experience and visualizations of waiting room traffic. It details the metrics available (Time in Queue, Time on Origin) and how they relate to waiting room settings (Total active users, New users per minute, Session duration, session renewal). It also explains how the analytics are visualized in the dashboard, comparing user traffic to configured thresholds. The post also touches on the underlying architecture of Waiting Room, mentioning its build on Cloudflare's Durable Objects.
  81. How Pingora keeps countIntroduced pingora-limits, a Rust library for Pingora, which implements the Count-Min Sketch algorithm for efficient, low-memory, and lock-free estimation of event rates and counts. This approach offers significant performance and memory advantages over traditional hash table implementations, especially at scale, by using O(log n) space and avoiding locks.
  82. How Cloudflare erroneously throttled a customer’s web trafficThis post details an incident where a customer's web traffic was erroneously throttled due to an unestablished process for handling high bandwidth consumption. The incident involved a sudden spike in traffic from a customer's domain, leading to congestion on a physical interface. A Cloudflare engineer mistakenly applied a throttling mechanism without proper sanction or established process. The throttle was lifted after 12 hours and 53 minutes. The post outlines corrective actions including establishing clear rules for mitigating such issues, requiring multiple levels of approval and customer communication for any action taken against a customer domain, improving tooling, and rewriting terms of service to better reflect current services and permitted uses.
  83. Back in 2017 we gave you Unmetered DDoS Mitigation, here's a birthday gift: Unmetered Rate Limiting for Self Serve customersIntroduced unmetered Rate Limiting for Free, Pro, and Business plans, removing usage-based charges. Launched a new Rate Limiting engine based on the ruleset engine, providing advanced rule-building capabilities similar to Custom Rules and Advanced Rate Limiting. This includes more fields for request matching (Path, Host, URI, Full URI, Query, Method, Source IP, User Agent), separate counting and mitigation expressions, and enhanced counting dimensions (IP, IP with NAT awareness, Query, Host, Headers, Cookie, ASN, Country, Path, JA3, JSON field). Increased counting periods for Pro (60 seconds) and Business (10 minutes) plans.
  84. Optimizing Magic Firewall’s IP listsThis post details the optimization of Magic Firewall's IP list storage by migrating from nftables sets to eBPF LPM Tries. This change significantly reduces memory consumption by sharing data across network namespaces and improves the performance of firewall rule updates by leveraging eBPF maps. The post also describes the integration of eBPF programs directly into the nftables configuration syntax, enabling atomic rule replacements and more advanced packet inspection capabilities.
  85. Introducing Advanced Rate LimitingIntroduces Advanced Rate Limiting, a new feature integrated with the Web Application Firewall (WAF) and Firewall Rules. This feature allows for more granular rate limiting by enabling counting requests based on virtually any HTTP request characteristic (URI, method, headers, cookies, body fields, bot score) and response characteristic (status code, response headers). It expands counting dimensions beyond IP to include country, header, cookie, AS Number (ASN), query parameter value, and JA3 fingerprint. It also introduces the concept of separating mitigation expressions from counting expressions and the ability to count based on request body content (JSON parsing). Future capabilities include complexity-based rate limiting.
  86. iCloud Private Relay: What Cloudflare Customers Need to KnowThis post details Cloudflare's role as a 'second relay' in Apple's iCloud Private Relay system. It explains how Private Relay splits connection metadata between Apple and Cloudflare to enhance user privacy. For website operators, it emphasizes the importance of updated geolocation databases and IPv6 reachability for accurate user location mapping. It also highlights that Cloudflare's existing rate limiting and bot management systems are equipped to handle traffic from Private Relay due to automatic multi-user IP detection and ML-based security heuristics.
  87. Waiting Room: Random Queueing and Custom Web/Mobile AppsIntroduced a new 'random queueing' method for Cloudflare Waiting Room, complementing the existing FIFO method. This new method allows users to be accepted into the application randomly, regardless of their arrival time, providing a fair chance for all participants in high-hype, short-lived scenarios. The implementation maintains the underlying FIFO structure, enabling seamless transitions between queueing methods by treating all users as 'new' for slot allocation when random queueing is active, while still respecting original bucket IDs and reserved slots for potential FIFO transitions. The post details the cookie properties (bucketId, acceptedAt, refreshIntervalSeconds, lastCheckInTime) and the allocation logic across data centers and Workers.
  88. Building Waiting Room on Workers and Durable ObjectsThis post details the technical design and implementation of Cloudflare Waiting Room, built on Cloudflare Workers and Durable Objects. It explains how the system manages user slots across data centers using historical traffic data and a distributed state management approach with Durable Objects. The cookie format and its contents (bucketId, lastCheckInTime, acceptedAt) are described, along with the session duration logic. The coordination mechanism between data centers using historical traffic ratios and the concept of bucketIds for fair queueing are elaborated.
  89. Announcing API Abuse DetectionThis post introduces API Discovery and API Abuse Detection. API Discovery maps API endpoints by normalizing paths and recording cardinality, reducing thousands of states to a manageable set. API Abuse Detection uses unsupervised learning for volumetric anomaly detection (adaptive rate limiting based on endpoint-specific baselines) and sequential anomaly detection (using Markov Chains to model state transitions and identify anomalous flows). These capabilities are applied to protect APIs and enhance Bot Management for mobile apps.
  90. Introducing Project Fair Shot: Ensuring COVID-19 Vaccine Registration Sites Can Keep Up With DemandThis post introduces Project Fair Shot, offering Cloudflare's Waiting Room service for free to organizations responsible for distributing COVID-19 vaccines. It details the problem of registration websites crashing under load due to high demand and explains how Waiting Room uses a queueing mechanism to manage traffic and prevent backend overload. The technical implementation of Waiting Room, which involves putting visitors into a queue ahead of the actual sale and then randomly shuffling and releasing them in batches, is described as a solution to fairly distribute scarce items.
  91. Cloudflare Waiting RoomThis post introduces Cloudflare Waiting Room, a new product built on Cloudflare's edge network and Workers. It addresses overwhelming demand for online resources by placing excess users into a virtual queue, protecting origin servers and ensuring user equity. The system leverages Workers and Durable Objects for a code-free setup, dynamically syncing with global demand to an application. Key configuration parameters include the number of active users, new users per minute, and session duration. The product offers default and customizable waiting room templates.
  92. How we made Firewall RulesThis post details the technical evolution of Cloudflare's configurable firewall, introducing Firewall Rules as a unified system for matching against multiple HTTP request attributes. It explains the limitations of previous single-attribute rules (IP, ASN, Country, User Agent, URI) and the challenges of combining them. The post introduces a Wireshark-inspired expression language for defining complex rules and discusses the technical choices, including the use of Rust for the new matching engine, to achieve real-time, high-performance rule evaluation.
  93. Rate Limiting: Delivering more rules, and greater controlThis post introduces new capabilities for Cloudflare Rate Limiting, including JavaScript Challenge and Google reCAPTCHA mitigation actions, and the ability to count on Origin Response Headers. It details how these features can be used to implement tiered protection strategies against various attacks like credential stuffing, enumeration attacks, and content scraping. The post also provides concrete examples of rule configurations for these use cases.
  94. Why does one NGINX worker take all the load?This post details the behavior of Linux's accept() system call with different socket configurations (single listen socket vs. SO_REUSEPORT) and its impact on load balancing across worker processes. It explains how the default epoll behavior can lead to uneven load distribution and how SO_REUSEPORT can improve balancing but potentially degrade latency distribution under high load. The post provides synthetic benchmarks and analysis to illustrate these trade-offs.
  95. How we built rate limiting capable of scaling to millions of domainsThis post details the implementation of a rate limiting system capable of scaling to millions of domains. It introduces a sliding window algorithm that uses memcached primitives (GET, SET, INCR) and asynchronous increment jobs to store counters per client IP address within each Point of Presence (PoP). The algorithm approximates the request rate by considering requests from the current and previous sampling periods. Mitigation information is cached locally on servers to avoid repeated memcached queries during high-traffic events or attacks.
  96. Cloudflare Rate Limiting - Insight, Control, and Mitigation against Layer 7 DDoS AttacksThis post announces the general availability of Cloudflare's Rate Limiting service to all customers, extending its protection against Layer 7 DDoS attacks, brute-force login attempts, and web scraping. It highlights real-world use cases and provides instructions for enabling and configuring the feature via the Cloudflare Dashboard. New capabilities include path-specific rules, HTTP method filtering for higher tiers, and a simulation mode for rule testing.
  97. The Porcupine Attack: investigating millions of junk requestsThis post details the investigation and analysis of a novel attack vector, dubbed 'The Porcupine Attack', which generated millions of junk HTTP requests per second. The attack bypassed traditional HTTP DDoS mitigation systems by sending malformed HTTP requests that were rejected by NGINX before mitigation could be applied. The investigation involved analyzing network traffic patterns, identifying a repeating sequence in the junk payload, and tracing the global distribution of attacking IP addresses. The analysis revealed a large, IP-rotating botnet with a bias towards South America and North Africa. The post also discusses the scale of the attack, the distribution of traffic across Cloudflare's Anycast network, and the unique characteristics of the attack that suggested a botnet rather than a legitimate connection attempt.
  98. Rate Limiting: Live DemoThis post introduces Cloudflare Rate Limiting, a product that allows customers to control traffic at the edge based on request rates per client IP, cookie, or other attributes. It details a live demonstration rule for cloudflare.com protecting `*.cloudflare.com/rate-limit-test` with a threshold of 10 requests per minute, blocking offending IPs for 1 minute with an HTML response. It also mentions future enhancements like method-specific rules, response code triggers, and additional matching criteria like headers and cookies, as well as mitigation options like CAPTCHA.
  99. Control your traffic at the edge with CloudflareIntroduced Cloudflare Rate Limiting, a new product allowing customers to control traffic based on requests per second per IP, cookie, or authentication token. Rules can be applied per-URI with wildcards. This feature aims to protect APIs and web pages from brute-force attacks, DDoS, and traffic floods, offering custom responses and fine-grained control over traffic flow to origin servers.
  100. Introducing the p0f BPF compilerThis post introduces the p0f BPF compiler, a tool that translates p0f SYN signatures into BPF bytecode. This enables the use of human-readable p0f signatures within iptables for high-speed packet discrimination and mitigation of SYN flood attacks. The post details the p0f signature format, provides examples of its use, and explains how to compile p0f signatures to BPF bytecode using the bpftools project.
  101. The curious case of slow downloadsThis post details the debugging and resolution of a specific issue where slow downloads were abruptly terminated due to a combination of Linux TCP send buffer behavior, NGINX's `send_timeout` setting, and the `reset_timedout_connection` option. The core problem was identified as the asymmetry in Linux socket writeability checks, where the buffer must drain below 66% of its capacity before being reported as writeable, leading to timeouts if the client download rate was too slow. The solution involved either increasing `send_timeout`, reducing `tcp_wmem`, or applying a Linux-specific NGINX patch to introduce a `send_minimum_rate` option.
  102. The Trouble with TorThis post details Cloudflare's challenges in distinguishing legitimate human traffic from malicious automated traffic originating from the Tor network. It explains how the high percentage of malicious requests from Tor exit nodes leads to frequent CAPTCHA challenges for Tor users. The post discusses the trade-offs between security, anonymity, and convenience, and explores potential solutions like treating GET requests differently and allowing customers to configure rules for Tor traffic. It also introduces open-sourced tools (torhoney and torexit) for analyzing Tor exit node reputation and longevity.
  103. The story of one latency spikeThis post details the debugging of a customer-reported latency issue on Cloudflare's CDN. The investigation used HTTP request timing, ICMP pings, tcpdump, and System Tap (stap) to identify the root cause as a kernel-level issue within the `tcp_collapse` function. The latency spikes were traced to the `sk_buff` metadata consuming excessive receive buffer space, leading to garbage collection stalls. The issue was mitigated by tuning the `net.ipv4.tcp_rmem` sysctl to a lower maximum value (4MiB), reducing `tcp_collapse` execution time and improving `net_rx_action` latency.
  104. Weird bug of the day: Twitter in-app browser can't visit siteIdentified and debugged an issue where the Twitter in-app browser's malformed Referer header (empty string when transitioning from HTTPS to HTTP) caused Cloudflare's Browser Integrity Check to block access. Implemented a fix to relax the Referer header validation for this specific client behavior, adhering to Postel's Law.
  105. Inside Shellshock: How hackers are using it to exploit systemsThis post details the analysis of the Shellshock vulnerability in the bash shell, explaining how it allows for arbitrary code execution via specially crafted HTTP headers. It describes Cloudflare's immediate rollout of WAF protection for Pro, Business, and Enterprise customers, followed by protection for Free plan customers. The post analyzes attack vectors observed through WAF logs, including reconnaissance, data exfiltration via email and file retrieval, and the use of the 'ping' command for identifying vulnerable machines. It also discusses the geographical origin of attacks and the techniques used by attackers.
  106. Introducing the BPF ToolsThis post introduces Cloudflare's open-sourced BPF Tools, designed for generating and deploying BPF rules to combat large-scale DDoS attacks, specifically focusing on DNS floods. It details the usage of tools like `pcap2hex`, `parsedns`, `bpfgen`, and `filter` for analyzing pcap files and generating BPF bytecode. The post explains how to create BPF rules for `xt_bpf` iptables module to drop malicious DNS requests targeting specific subdomains of `*.example.uk`. It also covers the generation of bash scripts using `iptables_bpf` and `iptables_bpf_chain` for easier deployment and management of these rules.
  107. Cloudflare: Happy 3rd BirthdayThe post describes the initial implementation of 'circuit breakers' to limit new signups during Cloudflare's launch. When unexpected surges occurred, the circuit breakers failed to shut down signups, leading to an unplanned scaling event. The engineering team had to monitor and manage the load manually, ultimately deciding to remove the circuit breakers based on confidence in the system's ability to scale.
  108. CloudFlare's new WAF: compiling to LuaThis post details the implementation of a new Web Application Firewall (WAF) written entirely in Lua, integrated with nginx via the HttpLuaModule. It describes the process of compiling rules from mod_security and a custom Cloudflare language into Lua code, including optimizations like clause reordering, regex simplification, and operator replacement. Performance was measured using line-level profiling (lulip) and flame graphs, leading to optimizations in LuaJIT and the WAF runtime. The goal of sub-1ms WAF decision times was achieved, with production metrics showing average processing times between 380us and 480us.
  109. How the CloudFlare Team Got Into Bondage (It's Not What You Think)This post details the implementation of port bonding (specifically 802.3ad Dynamic Link Aggregation) on servers to aggregate multiple 1Gbps network interfaces into a single logical interface with higher throughput. It also describes the manual configuration of IRQ affinity to isolate external and internal network traffic across different CPU cores, preventing SYN floods from impacting intra-server communication. This addresses bottlenecks in both raw bandwidth and network interrupt handling.
  110. Post Mortem: What Yesterday's Network Outage Looked LikeThis post details a network outage caused by a misapplied rate limit during a large DDoS attack. It highlights the need for automated smoke tests to prevent similar incidents in the future and discusses the relationship between upstream provider issues, DDoS attacks, and the impact of rate limiting on network availability. The post also provides a graph illustrating traffic patterns during the incident.
  111. CloudFlare Now Supporting More PortsThis post introduces support for proxying traffic on additional standard web ports (2052, 2053, 2082, 2083, 2086, 2087, 2095, 2096, 8080, 8443, 8880) beyond the initial 80 and 443. It clarifies that traffic on these new ports will not be cached or subject to performance/app transformations. It also explains the technical limitations preventing support for non-web protocols like FTP and SSH due to the lack of a HOST header, reinforcing the need to bypass Cloudflare for these protocols via IP address or a designated subdomain.
  112. Cloudflare Tips: Troubleshooting Common ProblemsThis post provides troubleshooting tips for common website issues encountered when using Cloudflare, including "Website is Unavailable" errors, issues with cPanel/FTP access, and problems with JavaScript (Rocket Loader) and CSS/image updates (Auto Minify, Development Mode). It details how to test server responsiveness using curl, explains how Cloudflare's reverse proxy affects access to cPanel/FTP, and offers guidance on managing Rocket Loader and Auto Minify. It also emphasizes the importance of allowlisting Cloudflare IPs and installing mod_cloudflare for proper visitor IP logging. Finally, it outlines how to report Cloudflare-specific issues and temporarily deactivate the service for debugging.
  113. Apache Killer Terminated: Zero Day Exploit, Zero Day FixCloudflare implemented a fix for the 'Apache Killer' zero-day exploit by filtering malformed or large numbers of byte-range headers from being relayed to origin servers. This protection was automatically applied to all Cloudflare users without requiring configuration changes. The fix was rolled out to half the network within hours and to the entire network within 24 hours.