DDoS Mitigation Architecture

4/11/2023

This post details Q1 2023 DDoS attack trends, including the rise of hyper-volumetric attacks (peaking at 71M rps), a 1.3 Tbps attack mitigated by Cloudflare, and the shift from IoT to VPS-based botnets. It highlights ongoing collaboration with cloud providers to dismantle botnets and the continued prevalence of Ransom DDoS attacks (16%). Key statistics include top targeted countries (Israel, US, Canada, Turkey for HTTP; China, Singapore, US, Finland for network-layer) and industries (Internet companies, Marketing and Advertising, Computer Software).

4/12/2022

This post details the DDoS attack trends observed in Q1 2022, highlighting a significant increase in application-layer attacks and volumetric network-layer attacks. It provides insights into attack sources, targets, and industries affected, including specific analysis of the Russian and Ukrainian cyberspace. The report also notes a decrease in ransom DDoS attacks compared to the previous quarter and year. The technical depth is primarily in the statistical analysis and reporting of attack vectors and trends, rather than specific architectural changes or new system builds.

8/5/2020

This post details the trends in network-layer DDoS attacks observed in Q2 2020, noting a doubling of L3/4 attacks compared to Q1, an increase in the scale of the largest attacks, and a wider variety of attack vectors. It highlights that while small attacks (under 10 Gbps) still dominate in number, large attacks are increasing in size. The post also details the dominance of SYN floods as an attack vector and the geographical distribution of attacks, with the US being the most targeted.

3/6/2018

This post details the fundamental role of IP spoofing in enabling large-scale L3 DDoS attacks, both direct and amplification. It explains why IP spoofing is possible due to internet design properties like multihoming, dynamic paths, and routing asymmetry. The post illustrates the difficulty of source IP filtering for receivers and the necessity of last-mile ISPs implementing BCP38. It also highlights how internet carriers can assist by cleaning their networks and aiding in investigations to identify spoofing customers during amplification attacks.

12/14/2017

This post provides a retrospective analysis of the Mirai IoT botnet, detailing its genesis, replication module (default credentials), and attack module (HTTP, UDP, TCP flooding). It highlights the scale of Mirai's impact, including attacks on Krebs on Security and OVH, and discusses the rise of copycat botnets. The analysis includes technical details on how Mirai propagates and its offensive capabilities, contributing to the understanding of large-scale IoT botnet threats and their role in DDoS attacks.

12/7/2017

This post announces Cloudflare's recognition as a leader in the Forrester Wave: DDoS Mitigation Solutions report. It attributes this leadership to Cloudflare's architecture designed for high-volume attacks, its unmetered mitigation offering (eliminating surge pricing), its ability to use automated mitigations for rapid detection and blocking of attacks (especially Layer 7), and the integration of DDoS mitigation with other security and performance features. The post references previous technical deep dives into the architecture and automated mitigation strategies.

11/23/2017

This post details the shift in DDoS attack strategies from network-layer floods to more sophisticated application-layer attacks. It explains how botnets, IoT devices, and DNS amplification are used to build attacker capacity. The post contrasts older DDoS mitigation priorities (capacity, Anycast network) with the new landscape, highlighting the decreasing frequency of large network-level attacks and the increasing prevalence of application-layer attacks. It also mentions the introduction of 'Unmetered Mitigation' as a policy change.

9/25/2017

This post details Cloudflare's architectural shift away from dedicated scrubbing centers for DDoS mitigation. It highlights the problems with traditional scrubbing (bandwidth, cost, knowledge gap) and presents Cloudflare's alternative: integrating mitigation into its global network and servers. Key technical aspects include leveraging commodity hardware, an anycast network for distributed attack handling, iptables with modules like xt_bpf, ipset, and hashlimits, and the use of Solarflare EFVI for kernel bypass to offload firewall rules to user space for high-packet-per-second processing. The post also emphasizes the benefits of this approach for improving the overall software stack and the company's commitment to open-sourcing DDoS mitigation tools.

9/25/2017

This post introduces Gatebot, an automated mitigation system for DDoS attacks. Gatebot consists of three parts: attack detection (using Golang and streaming algorithms), reactive automation (business logic in Python), and mitigation deployment. It automates the workflow of observing network anomalies, identifying attack targets, and applying appropriate mitigation actions, such as scattering DNS resolutions, using iptables with extensions like xt_bpf, ipsets, hashlimits, and connlimit, and employing kernel bypass user space programs (floodgate) for high-packet-per-second attacks. Gatebot has significantly improved the speed and precision of mitigation compared to manual handling by SREs, allowing for more confident operation and enabling the same level of DDoS protection for all customers.

8/28/2017

This post details the collaborative effort to disrupt the WireX botnet, which comprised primarily Android devices and launched application-layer DDoS attacks using HTTP GET requests with distinctive, randomized User-Agent strings. The collaboration involved multiple CDNs, content providers, and intelligence firms, highlighting the importance of information sharing and joint efforts in combating large-scale botnets.

2/6/2017

This post details a new DDoS ransom group that differs from previous copycats by sending a small demonstration attack before the ransom email. The demonstration attacks typically peak around 10 Gbps, last for less than an hour, and use DNS amplification or NTP reflection. The post also notes that the group does not launch a second attack if the ransom is not paid and that there is no evidence linking these attacks to the Mirai botnet, despite references in some ransom emails. It advises against paying ransoms and encourages proactive DDoS protection.

12/2/2016

This post details a ten-day period of massive, sustained L3/L4 TCP flood attacks, peaking at 480Gbps and 172Mpps, which were automatically mitigated without customer impact. It highlights the attackers' shift to working regular hours and the use of different attack software than the Mirai botnet, with attacks concentrated on the US west coast. The post reinforces the effectiveness of Cloudflare's always-on, distributed DDoS mitigation architecture.

10/26/2016

This post details Cloudflare's architectural approach to scaling DDoS mitigation, contrasting it with legacy 'scrubbing center' models. It highlights the use of commodity hardware, BGP for load distribution within and across data centers, and the ability for every server to handle any request. The economic advantage of this distributed model is explained through Cloudflare's bandwidth purchasing strategy (95th percentile billing) and how caching traffic naturally offsets ingress attack traffic, making DDoS mitigation effectively free. The post emphasizes the continuous arms race with attackers and Cloudflare's position to win due to its scalable, cost-effective, and distributed architecture.

10/11/2016

This post details the analysis of large-scale DDoS attacks originating from IoT botnets, specifically highlighting a shift towards L7 HTTP-based attacks. It presents data on two distinct attack types: one peaking at 1.75 Mrps with short HTTP requests and another generating 360Gbps with long HTTP request payloads. The analysis includes source IP distribution across AS numbers, datacenter concentration, and evidence suggesting the use of compromised IoT devices like CCTV cameras and NAS devices. It also discusses the future of DDoS threats from an increasing number of internet-connected devices.

8/2/2016

This post introduces the open-sourcing of Cloudflare's p0f BPF compiler, a component of their bpftools. The compiler translates p0f SYN signatures into BPF bytecode, allowing iptables to efficiently discriminate and mitigate attack packets based on their signatures. The post details the p0f signature format, provides examples of its use in distinguishing legitimate traffic from attack traffic, and explains how to compile p0f signatures to BPF bytecode using the bpftools. It also includes an example of blocking hping3 generated SYN packets.

4/25/2016

This post analyzes the threat landscape of extortion-based DDoS threats, specifically the Armada Collective. It details the consistent extortion emails received by over 100 customers, the requested Bitcoin amounts, and the lack of actual DDoS attacks launched by this specific incarnation of the group. The post also contrasts this with the original Armada Collective (suspected to be DD4BC) which did launch attacks, and highlights Cloudflare's successful mitigation of those. It emphasizes that while many threats are empty, the company remains prepared to mitigate actual attacks and provides guidance on how to respond to such threats.

3/3/2016

This post details the scale of recent Layer 3 DDoS attacks, peaking at 400Gbps and 180Mpps. It highlights Cloudflare's ability to absorb and accurately measure these attacks due to its network capacity and automated mitigation systems, avoiding the need for BGP blackholing. The post also discusses the challenges of measuring attacks when blackholing is employed and presents metrics on packet rates and bandwidth during these events.

9/25/2015

This post details a specific instance of a browser-based DDoS attack that leveraged mobile ad networks as a distribution vector. The attack involved malicious JavaScript embedded in advertisements, which instructed mobile devices to issue a large volume of HTTP POST requests to a victim's site. Cloudflare analyzed the attack's characteristics, including the request patterns, user agents, and origin country, and theorized the distribution mechanism through ad networks. The post highlights the growing threat of browser-based floods and Cloudflare's ability to handle them.

9/7/2015

This post introduces and analyzes various kernel bypass techniques for high-speed packet processing, addressing the limitations of vanilla Linux networking for handling large packet floods. It details experiments demonstrating the performance drop of the Linux kernel stack with increasing CPU cores and explores Packet_mmap, PF_RING, Snabbswitch, DPDK, Netmap, Solarflare's EF_VI, bifurcated drivers, and virtualization approaches (VF interfaces) as solutions. The post highlights the challenges of dedicating entire NICs to bypass applications and the specific advantages of Solarflare's EF_VI and the bifurcated driver concept.

4/30/2015

This post introduces JavaScript-based DDoS attacks, detailing how malicious JavaScript embedded in websites or compromised third-party scripts can turn unwitting users into participants. It explains the mechanics of Layer 7 attacks using JavaScript, the risks associated with compromised third-party scripts (e.g., jQuery), and introduces Subresource Integrity (SRI) as a defense mechanism. It also highlights HTTPS as a complete solution against man-in-the-middle injection of malicious JavaScript.

7/3/2014

This post introduces Cloudflare's open-sourced BPF Tools, which include scripts for analyzing pcap files, parsing DNS traffic, generating BPF bytecode, and deploying it via iptables. The tools focus on filtering DNS traffic and mitigating stateless floods, specifically demonstrating how to drop packets targeting subdomains of a given domain using the `xt_bpf` iptables module. The post details the usage of `pcap2hex`, `parsedns`, `bpfgen`, `filter`, `iptables_bpf`, and `iptables_bpf_chain` for generating and verifying BPF rules to combat DDoS attacks.

3/12/2014

This post details the Texas Tribune's experience implementing Cloudflare to handle a significant traffic spike and potential security threats during the livestreaming of interviews with Julian Assange and Edward Snowden at SXSW. The implementation involved setting up Cloudflare to defend against DDoS attacks and speed up content delivery, particularly for a static site hosted on Amazon S3. The successful handling of a 10X traffic spike from over 200 countries without downtime or security issues is highlighted.

7/30/2013

This post details how Cloudflare's DNS acts as a proxy, directing traffic to Cloudflare's network instead of the origin server. It emphasizes the importance of keeping the origin IP secret by advising users to: keep all subdomains on Cloudflare, remove wildcard subdomains for non-enterprise plans, review DNS records for IP mentions, avoid hosting mail/other services on the same server as the web server, never initiate outbound connections based on user action, patch information disclosure vulnerabilities, and change the origin IP after configuration. It also links to general website security guidelines and a DDoS attack response document.

3/27/2013

This post details a massive DDoS attack against Spamhaus, peaking at 120Gbps, which Cloudflare mitigated using its Anycast network. The attack evolved to target Cloudflare's upstream bandwidth providers and Internet Exchanges (IXs), demonstrating the potential for such attacks to impact core internet infrastructure and cause widespread congestion. The post highlights the vulnerability of open DNS recursors as a source for amplification attacks and discusses best practices for IX security.

3/20/2013

This post details Cloudflare's mitigation of a large DDoS attack against Spamhaus, which reached approximately 75Gbps. The primary attack vector was DNS reflection using open DNS resolvers, with a 100x amplification factor. Cloudflare's Anycast network diluted the attack across its global data centers, and custom software dropped unmatched ACKs from ACK reflection attacks. The post also touches on the historical context of fighting spam via DNSBLs and the ongoing battle against open DNS resolvers.

3/3/2013

This post details a system-wide outage caused by a bug in the Flowspec implementation on Juniper routers. A misconfigured rule, intended to drop large packets during a DDoS attack, instead caused all edge routers to consume all their RAM and crash. The incident response involved identifying the bad rule, removing it, and physically rebooting unresponsive routers. This incident led to plans for more extensive testing of Flowspec provisioned filters and evaluating ways to isolate rule application to specific data centers.

2/14/2013

This post details the evolution of Cloudflare's 'I'm Under Attack Mode' (IUAM) from a simple 2+2 math challenge to a more sophisticated JavaScript-based challenge designed to thwart malware like 'OutFlare'. It also outlines plans to implement proof-of-work problems for future enhancements. The post emphasizes the advantage of Cloudflare's cloud-based service for real-time adaptation to new threats.

11/28/2012

This post details the increase in Layer 7 (application-layer) attacks observed during a ceasefire in a Middle Eastern conflict. It highlights Cloudflare's ability to absorb these attacks and ensure legitimate requests reach web servers, differentiating them from Layer 3/4 attacks. The post also discusses the political implications of acting as a proxy for both sides of a conflict and reiterates Cloudflare's commitment to keeping all websites online.

9/16/2012

This post details a specific incident where a misapplied network rate limit, intended to mitigate a large DDoS attack, caused a widespread outage in Europe. It describes the two visible events: an upstream bandwidth provider issue and the misapplied rate limit. The post explains the technical details of the DDoS attack (layer 4, 65 Gbps) and the subsequent mistake in applying a rate limit, which impacted a wider customer base than intended due to concentrated traffic in Europe. It also mentions the need for additional smoke tests to prevent similar future incidents and highlights the importance of control plane resilience.

8/16/2012

This post analyzes Layer 7 HTTP DoS attacks against Cloudflare sites, detailing their prevalence (95.5% of attacks), origin (traceable IP addresses), and temporal patterns. It highlights a significant increase in attacks on Saturdays and a general upward trend in Layer 7 DoS activity over the first half of 2012, contrasting with a slight decline in lower-level DDoS attacks. The post also identifies the top originating countries and ISPs for these attacks and discusses Cloudflare's ongoing efforts to defend against them.

8/3/2012

This post provides early statistics on DoS attacks observed by Cloudflare, detailing attack frequency, timing (day of week, time of day), protocol distribution (UDP vs. TCP), common target ports (HTTP, DNS), and specific attack vectors like SYN flooding and reflection attacks. It also notes the impact of DNSSEC on certain attacks and the emergence of 'carpet bombing' and short-duration 'booter' attacks.

12/22/2011

This post identifies and categorizes the rise of DDoS attacks in 2011, noting a 700% increase and attributing them to political and financial motivations. It highlights Cloudflare's role in keeping sites online during attacks.

8/25/2011

Implemented a network-wide fix for the 'Apache Killer' zero-day exploit by filtering malformed or excessive byte-range headers before they reach origin servers. This protection was automatically applied to all Cloudflare users without requiring configuration changes.